Wearable inward-facing camera utilizing the Scheimpflug principle

ABSTRACT

One aspect of this disclosure involves a wearable device that includes a frame that is worn on a user&#39;s head, and an inward-facing camera (camera) physically coupled to the frame. The optical axis of the camera is either above the Frankfort horizontal plane and pointed upward to capture an image of a region of interest (ROI) above the user&#39;s eyes, or the optical axis is below the Frankfort horizontal plane and pointed downward to capture an image of an ROI below the user&#39;s eyes. The camera includes a sensor and a lens. The sensor plane is tilted by more than 2° relative to the lens plane according to the Scheimpflug principle in order to capture a sharper image. The Scheimpflug principle is a geometric rule that describes the orientation of the plane of focus of a camera when the lens plane is tilted relative to the sensor plane.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/408,677, filed Oct. 14, 2016, and U.S. Provisional PatentApplication No. 62/456,105, filed Feb. 7, 2017, and U.S. ProvisionalPatent Application No. 62/480,496, filed Apr. 2, 2017.

This application is a Continuation-In-Part of U.S. application Ser. No.15/182,592, filed Jun. 14, 2016, and a Continuation-In-Part of U.S.application Ser. No. 15/231,276, filed Aug. 8, 2016, and aContinuation-In-Part of U.S. application Ser. No. 15/284,528, filed Oct.3, 2016.

BACKGROUND

When mounting a camera having a large field of view in sharp angle andclose to the face, the captured image is usually not sharp all overbecause the object is not parallel to the sensor plane. There is a needto improve the quality of the images obtained from a camera mounted inclose proximity and sharp angle to the face.

SUMMARY

Normally, the lens plane and the sensor plane of a camera are parallel,and the plane of focus (PoF) is parallel to the lens and sensor planes.If a planar object is also parallel to the sensor plane, it can coincidewith the PoF, and the entire object can be captured sharply. If the lensplane is tilted (not parallel) relative to the sensor plane, it will bein focus along a line where it intersects the PoF. The Scheimpflugprinciple is a known geometric rule that describes the orientation ofthe plane of focus of a camera when the lens plane is tilted relative tothe sensor plane.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are herein described by way of example only, withreference to the following drawings:

FIG. 1a and FIG. 1b illustrate various inward-facing head-mountedcameras coupled to an eyeglasses frame;

FIG. 2 illustrates inward-facing head-mounted cameras coupled to anaugmented reality device;

FIG. 3 illustrates head-mounted cameras coupled to a virtual realitydevice;

FIG. 4 illustrates a side view of head-mounted cameras coupled to anaugmented reality device;

FIG. 5 illustrates a side view of head-mounted cameras coupled to asunglasses frame;

FIG. 6 to FIG. 9 illustrate HMSs configured to measure various ROIsrelevant to some of the embodiments describes herein;

FIG. 10 to FIG. 13 illustrate various embodiments of systems thatinclude inward-facing head-mounted cameras having multi-pixel sensors(FPA sensors);

FIG. 14a , FIG. 14b , and FIG. 14c illustrate embodiments of two rightand left clip-on devices that re configured to attached/detached from aneyeglasses frame;

FIG. 15a and FIG. 15b illustrate an embodiment of a clip-on device thatincludes inward-facing head-mounted cameras pointed at the lower part ofthe face and the forehead;

FIG. 16a and FIG. 16b illustrate embodiments of right and left clip-ondevices that are configured to be attached behind an eyeglasses frame;

FIG. 17a and FIG. 17b illustrate an embodiment of a single-unit clip-ondevice that is configured to be attached behind an eyeglasses frame;

FIG. 18 illustrates embodiments of right and left clip-on devices, whichare configured to be attached/detached from an eyeglasses frame, andhave protruding arms to hold inward-facing head-mounted cameras;

FIG. 19 illustrates a scenario in which an alert regarding a possiblestroke is issued;

FIG. 20a is a schematic illustration of an inward-facing head-mountedcamera embedded in an eyeglasses frame, which utilizes the Scheimpflugprinciple;

FIG. 20b is a schematic illustration of a camera that is able to changethe relative tilt between its lens and sensor planes according to theScheimpflug principle;

FIG. 21a and FIG. 21b illustrate a scenario in which a user is alertedabout an expected allergic reaction;

FIG. 22 illustrates a scenario in which a trigger of an allergicreaction may be identified;

FIG. 23 illustrates an embodiment of an HMS able to measure stresslevel;

FIG. 24 illustrates examples of asymmetric locations of inward-facinghead-mounted thermal cameras (CAMs) that measure the periorbital areas;

FIG. 25 illustrates an example of symmetric locations of the CAMs thatmeasure the periorbital areas;

FIG. 26 illustrates a scenario in which a system suggests to the user totake a break in order to reduce the stress level;

FIG. 27a illustrates a child watching a movie while wearing aneyeglasses frame with at least five CAMs;

FIG. 27b illustrates generation of a graph of the stress level of thechild detected at different times while different movie scenes wereviewed;

FIG. 28 illustrates an embodiment of a system that generates apersonalized model for detecting stress based on thermal measurements ofthe face;

FIG. 29 illustrates an embodiment of a system that includes a userinterface, which notifies a user when the stress level of the userreaches a predetermined threshold;

FIG. 30 illustrates an embodiment of a system that selects a stressor;and

FIG. 31a and FIG. 31b are schematic illustrations of embodiments ofcomputers.

DETAILED DESCRIPTION

A “thermal camera” refers herein to a non-contact device that measureselectromagnetic radiation having wavelengths longer than 2500 nanometer(nm) and does not touch its region of interest (ROI). A thermal cameramay include one sensing element (pixel), or multiple sensing elementsthat are also referred to herein as “sensing pixels”, “pixels”, and/orfocal-plane array (FPA). A thermal camera may be based on an uncooledthermal sensor, such as a thermopile sensor, a microbolometer sensor(where microbolometer refers to any type of a bolometer sensor and itsequivalents), a pyroelectric sensor, or a ferroelectric sensor.

Sentences in the form of “thermal measurements of an ROI” (usuallydenoted TH_(ROI) or some variant thereof) refer to at least one of: (i)temperature measurements of the ROI (T_(ROI)), such as when usingthermopile or microbolometer sensors, and (ii) temperature changemeasurements of the ROI (ΔT_(ROI)), such as when using a pyroelectricsensor or when deriving the temperature changes from temperaturemeasurements taken at different times by a thermopile sensor or amicrobolometer sensor.

In some embodiments, a device, such as a thermal camera, may bepositioned such that it occludes an ROI on the user's face, while inother embodiments, the device may be positioned such that it does notocclude the ROI. Sentences in the form of “the system/camera does notocclude the ROI” indicate that the ROI can be observed by a third personlocated in front of the user and looking at the ROI, such as illustratedby all the ROIs in FIG. 7, FIG. 11 and FIG. 19. Sentences in the form of“the system/camera occludes the ROI” indicate that some of the ROIscannot be observed directly by that third person, such as ROIs 19 and 37that are occluded by the lenses in FIG. 1a , and ROIs 97 and 102 thatare occluded by cameras 91 and 96, respectively, in FIG. 9.

Although many of the disclosed embodiments can use occluding thermalcameras successfully, in certain scenarios, such as when using an HMS ona daily basis and/or in a normal day-to-day setting, using thermalcameras that do not occlude their ROIs on the face may provide one ormore advantages to the user, to the HMS, and/or to the thermal cameras,which may relate to one or more of the following: esthetics, betterventilation of the face, reduced weight, simplicity to wear, and reducedlikelihood to being tarnished.

A “Visible-light camera” refers to a non-contact device designed todetect at least some of the visible spectrum, such as cameras withoptical lenses and CMOS or CCD sensors.

The term “inward-facing head-mounted camera” refers to a cameraconfigured to be worn on a user's head and to remain pointed at its ROI,which is on the user's face, also when the user's head makes angular andlateral movements (such as movements with an angular velocity above 0.1rad/sec, above 0.5 rad/sec, and/or above 1 rad/sec). A head-mountedcamera (which may be inward-facing and/or outward-facing) may bephysically coupled to a frame worn on the user's head, may be attachedto eyeglass using a clip-on mechanism (configured to be attached to anddetached from the eyeglasses), or may be mounted to the user's headusing any other known device that keeps the camera in a fixed positionrelative to the user's head also when the head moves. Sentences in theform of “camera physically coupled to the frame” mean that the cameramoves with the frame, such as when the camera is fixed to (or integratedinto) the frame, or when the camera is fixed to (or integrated into) anelement that is physically coupled to the frame. The abbreviation “CAM”denotes “inward-facing head-mounted thermal camera”, the abbreviation“CAM_(out)” denotes “outward-facing head-mounted thermal camera”, theabbreviation “VCAM” denotes “inward-facing head-mounted visible-lightcamera”, and the abbreviation “VCAM_(out)” denotes “outward-facinghead-mounted visible-light camera”.

Sentences in the form of “a frame configured to be worn on a user'shead” or “a frame worn on a user's head” refer to a mechanical structurethat loads more than 50% of its weight on the user's head. For example,an eyeglasses frame may include two temples connected to two rimsconnected by a bridge; the frame in Oculus Rift™ includes the foamplaced on the user's face and the straps; and the frames in GoogleGlass™ and Spectacles by Snap Inc. are similar to eyeglasses frames.Additionally or alternatively, the frame may connect to, be affixedwithin, and/or be integrated with, a helmet (e.g., sports, motorcycle,bicycle, and/or combat helmets) and/or a brainwave-measuring headset.

When a thermal camera is inward-facing and head-mounted, challengesfaced by systems known in the art that are used to acquire thermalmeasurements, which include non-head-mounted thermal cameras, may besimplified and even eliminated with some of the embodiments describedherein. Some of these challenges may involve dealing with complicationscaused by movements of the user, image registration, ROI alignment,tracking based on hot spots or markers, and motion compensation in theIR domain.

In various embodiments, cameras are located close to a user's face, suchas at most 2 cm, 5 cm, 10 cm, 15 cm, or 20 cm from the face (herein “cm”denotes to centimeters). The distance from the face/head in sentencessuch as “a camera located less than 15 cm from the face/head” refers tothe shortest possible distance between the camera and the face/head. Thehead-mounted cameras used in various embodiments may be lightweight,such that each camera weighs below 10 g, 5 g, 1 g, and/or 0.5 g (herein“g” denotes to grams).

The following figures show various examples of HMSs equipped withhead-mounted cameras. FIG. 1a illustrates various inward-facinghead-mounted cameras coupled to an eyeglasses frame 15. Cameras 10 and12 measure regions 11 and 13 on the forehead, respectively. Cameras 18and 36 measure regions on the periorbital areas 19 and 37, respectively.The HMS further includes an optional computer 16, which may include aprocessor, memory, a battery and/or a communication module. FIG. 1billustrates a similar HMS in which inward-facing head-mounted cameras 48and 49 measure regions 41 and 41, respectively. Cameras 22 and 24measure regions 23 and 25, respectively. Camera 28 measures region 29.And cameras 26 and 43 measure regions 38 and 39, respectively.

FIG. 2 illustrates inward-facing head-mounted cameras coupled to anaugmented reality device such as Microsoft HoloLens™. FIG. 3 illustrateshead-mounted cameras coupled to a virtual reality device such asFacebook's Oculus Rift™. FIG. 4 is a side view illustration ofhead-mounted cameras coupled to an augmented reality device such asGoogle Glass™. FIG. 5 is another side view illustration of head-mountedcameras coupled to a sunglasses frame.

FIG. 6 to FIG. 9 illustrate HMSs configured to measure various ROIsrelevant to some of the embodiments describes herein. FIG. 6 illustratesa frame 35 that mounts inward-facing head-mounted cameras 30 and 31 thatmeasure regions 32 and 33 on the forehead, respectively. FIG. 7illustrates a frame 75 that mounts inward-facing head-mounted cameras 70and 71 that measure regions 72 and 73 on the forehead, respectively, andinward-facing head-mounted cameras 76 and 77 that measure regions 78 and79 on the upper lip, respectively. FIG. 8 illustrates a frame 84 thatmounts inward-facing head-mounted cameras 80 and 81 that measure regions82 and 83 on the sides of the nose, respectively. And FIG. 9 illustratesa frame 90 that includes (i) inward-facing head-mounted cameras 91 and92 that are mounted to protruding arms and measure regions 97 and 98 onthe forehead, respectively, (ii) inward-facing head-mounted cameras 95and 96, which are also mounted to protruding arms, which measure regions101 and 102 on the lower part of the face, respectively, and (iii)head-mounted cameras 93 and 94 that measure regions on the periorbitalareas 99 and 100, respectively.

FIG. 10 to FIG. 13 illustrate various inward-facing head-mounted camerashaving multi-pixel sensors (FPA sensors), configured to measure variousROIs relevant to some of the embodiments describes herein. FIG. 10illustrates head-mounted cameras 120 and 122 that measure regions 121and 123 on the forehead, respectively, and mounts head-mounted camera124 that measure region 125 on the nose. FIG. 11 illustrateshead-mounted cameras 126 and 128 that measure regions 127 and 129 on theupper lip, respectively, in addition to the head-mounted cameras alreadydescribed in FIG. 10. FIG. 12 illustrates head-mounted cameras 130 and132 that measure larger regions 131 and 133 on the upper lip and thesides of the nose, respectively. And FIG. 13 illustrates head-mountedcameras 134 and 137 that measure regions 135 and 138 on the right andleft cheeks and right and left sides of the mouth, respectively, inaddition to the head-mounted cameras already described in FIG. 12.

In some embodiments, the head-mounted cameras may be physically coupledto the frame using a clip-on device configured to be attached/detachedfrom a pair of eyeglasses in order to secure/release the device to/fromthe eyeglasses, multiple times. The clip-on device holds at least aninward-facing camera, a processor, a battery, and a wirelesscommunication module. Most of the clip-on device may be located in frontof the frame (as illustrated in FIG. 14b . FIG. 15b , and FIG. 18), oralternatively, most of the clip-on device may be located behind theframe, as illustrated in FIG. 16b and FIG. 17 b.

FIG. 14a FIG. 14b , and FIG. 14c illustrate two right and left clip-ondevices 141 and 142, respectively, configured to attached/detached froman eyeglasses frame 140. The clip-on device 142 includes aninward-facing head-mounted camera 143 pointed at a region on the lowerpart of the face (such as the upper lip, mouth, nose, and/or cheek), aninward-facing head-mounted camera 144 pointed at the forehead, and otherelectronics 145 (such as a processor, a battery, and/or a wirelesscommunication module). The clip-on devices 141 and 142 may includeadditional cameras illustrated in the drawings as black circles.

FIG. 15a and FIG. 15b illustrate a clip-on device 147 that includes aninward-facing head-mounted camera 148 pointed at a region on the lowerpart of the face (such as the nose), and an inward-facing head-mountedcamera 149 pointed at the forehead. The other electronics (such as aprocessor, a battery, and/or a wireless communication module) is locatedinside the box 150, which also holds the cameras 148 and 149.

FIG. 16a and FIG. 16b illustrate two right and left clip-on devices 160and 161, respectively, configured to be attached behind an eyeglassesframe 165. The clip-on device 160 includes an inward-facing head-mountedcamera 162 pointed at a region on the lower part of the face (such asthe upper lip, mouth, nose, and/or cheek), an inward-facing head-mountedcamera 163 pointed at the forehead, and other electronics 164 (such as aprocessor, a battery, and/or a wireless communication module). Theclip-on devices 160 and 161 may include additional cameras illustratedin the drawings as black circles.

FIG. 17a and FIG. 17b illustrate a single-unit clip-on device 170,configured to be attached behind an eyeglasses frame 176. Thesingle-unit clip-on device 170 includes inward-facing head-mountedcameras 171 and 172 pointed at regions on the lower part of the face(such as the upper lip, mouth, nose, and/or cheek), inward-facinghead-mounted cameras 173 and 174 pointed at the forehead, a spring 175configured to apply force that holds the clip-on device 170 to the frame176, and other electronics 177 (such as a processor, a battery, and/or awireless communication module). The clip-on device 170 may includeadditional cameras illustrated in the drawings as black circles.

FIG. 18 illustrates two right and left clip-on devices 153 and 154,respectively, configured to attached/detached from an eyeglasses frame,and having protruding arms to hold the inward-facing head-mountedcameras. Head-mounted camera 155 measures a region on the lower part ofthe face, head-mounted camera 156 measures regions on the forehead, andthe left clip-on device 154 further includes other electronics 157 (suchas a processor, a battery, and/or a wireless communication module). Theclip-on devices 153 and 154 may include additional cameras illustratedin the drawings as black circles.

It is noted that the elliptic and other shapes of the ROIs in some ofthe drawings are just for illustration purposes, and the actual shapesof the ROIs are usually not as illustrated. It is possible to calculatethe accurate shape of an ROI using various methods, such as acomputerized simulation using a 3D model of the face and a model of ahead-mounted system (HMS) to which a thermal camera is physicallycoupled, or by placing a LED instead of the sensor (while maintainingthe same field of view) and observing the illumination pattern on theface. Furthermore, illustrations and discussions of a camera representone or more cameras, where each camera may have the same FOV and/ordifferent FOVs. Unless indicated to the contrary, the cameras mayinclude one or more sensing elements (pixels), even when multiplesensing elements do not explicitly appear in the figures; when a cameraincludes multiple sensing elements then the illustrated ROI usuallyrefers to the total ROI captured by the camera, which is made ofmultiple regions that are respectively captured by the different sensingelements. The positions of the cameras in the figures are just forillustration, and the cameras may be placed at other positions on theHMS.

Sentences in the form of an “ROI on an area”, such as ROI on theforehead or an ROI on the nose, refer to at least a portion of the area.Depending on the context, and especially when using a CAM having justone pixel or a small number of pixels, the ROI may cover another area(in addition to the area). For example, a sentence in the form of “anROI on the nose” may refer to either: 100% of the ROI is on the nose, orsome of the ROI is on the nose and some of the ROI is on the upper lip.

Various embodiments described herein involve detections of physiologicalresponses based on user measurements. Some examples of physiologicalresponses include stress, an allergic reaction, an asthma attack, astroke, dehydration, intoxication, or a headache (which includes amigraine). Other examples of physiological responses includemanifestations of fear, startle, sexual arousal, anxiety, joy, pain orguilt. Still other examples of physiological responses includephysiological signals such as a heart rate or a value of a respiratoryparameter of the user. Optionally, detecting a physiological responsemay involve one or more of the following: determining whether the userhas/had the physiological response, identifying an imminent attackassociated with the physiological response, and/or calculating theextent of the physiological response.

In some embodiments, detection of the physiological response is done byprocessing thermal measurements that fall within a certain window oftime that characterizes the physiological response. For example,depending on the physiological response, the window may be five secondslong, thirty seconds long, two minutes long, five minutes long, fifteenminutes long, or one hour long. Detecting the physiological response mayinvolve analysis of thermal measurements taken during multiple of theabove-described windows, such as measurements taken during differentdays. In some embodiments, a computer may receive a stream of thermalmeasurements, taken while the user wears an HMS with coupled thermalcameras during the day, and periodically evaluate measurements that fallwithin a sliding window of a certain size.

In some embodiments, models are generated based on measurements takenover long periods. Sentences of the form of “measurements taken duringdifferent days” or “measurements taken over more than a week” are notlimited to continuous measurements spanning the different days or overthe week, respectively. For example, “measurements taken over more thana week” may be taken by eyeglasses equipped with thermal cameras, whichare worn for more than a week, 8 hours a day. In this example, the useris not required to wear the eyeglasses while sleeping in order to takemeasurements over more than a week. Similarly, sentences of the form of“measurements taken over more than 5 days, at least 2 hours a day” referto a set comprising at least 10 measurements taken over 5 differentdays, where at least two measurements are taken each day at timesseparated by at least two hours.

Utilizing measurements taken of a long period (e.g., measurements takenon “different days”) may have an advantage, in some embodiments, ofcontributing to the generalizability of a trained model. Measurementstaken over the long period likely include measurements taken indifferent environments and/or measurements taken while the measured userwas in various physiological and/or mental states (e.g., before/aftermeals and/or while the measured user wassleepy/energetic/happy/depressed, etc.). Training a model on such datacan improve the performance of systems that utilize the model in thediverse settings often encountered in real-world use (as opposed tocontrolled laboratory-like settings). Additionally, taking themeasurements over the long period may have the advantage of enablingcollection of a large amount of training data that is required for somemachine learning approaches (e.g., “deep learning”).

Detecting the physiological response may involve performing varioustypes of calculations by a computer. Optionally, detecting thephysiological response may involve performing one or more of thefollowing operations: comparing thermal measurements to a threshold(when the threshold is reached that may be indicative of an occurrenceof the physiological response), comparing thermal measurements to areference time series, and/or by performing calculations that involve amodel trained using machine learning methods. Optionally, the thermalmeasurements upon which the one or more operations are performed aretaken during a window of time of a certain length, which may optionallydepend on the type of physiological response being detected. In oneexample, the window may be shorter than one or more of the followingdurations: five seconds, fifteen seconds, one minute, five minutes,thirty minute, one hour, four hours, one day, or one week. In anotherexample, the window may be longer than one or more of the aforementioneddurations. Thus, when measurements are taken over a long period, such asmeasurements taken over a period of more than a week, detection of thephysiological response at a certain time may be done based on a subsetof the measurements that falls within a certain window near the certaintime; the detection at the certain time does not necessarily involveutilizing all values collected throughout the long period.

In some embodiments, detecting the physiological response of a user mayinvolve utilizing baseline thermal measurement values, most of whichwere taken when the user was not experiencing the physiologicalresponse. Optionally, detecting the physiological response may rely onobserving a change to typical temperatures at one or more ROIs (thebaseline), where different users might have different typicaltemperatures at the ROIs (i.e., different baselines). Optionally,detecting the physiological response may rely on observing a change to abaseline level, which is determined based on previous measurements takenduring the preceding minutes and/or hours.

In some embodiments, detecting a physiological response involvesdetermining the extent of the physiological response, which may beexpressed in various ways that are indicative of the extent of thephysiological response, such as: (i) a binary value indicative ofwhether the user experienced, and/or is experiencing, the physiologicalresponse, (ii) a numerical value indicative of the magnitude of thephysiological response, (iii) a categorial value indicative of theseverity/extent of the physiological response, (iv) an expected changein thermal measurements of an ROI (denoted TH_(ROI) or some variationthereof), and/or (v) rate of change in TH_(ROI). Optionally, when thephysiological response corresponds to a physiological signal (e.g., aheart rate, a breathing rate, and an extent of frontal lobe brainactivity), the extent of the physiological response may be interpretedas the value of the physiological signal.

Herein, “machine learning” methods refers to learning from examplesusing one or more approaches. Optionally, the approaches may beconsidered supervised, semi-supervised, and/or unsupervised methods.Examples of machine learning approaches include: decision tree learning,association rule learning, regression models, nearest neighborsclassifiers, artificial neural networks, deep learning, inductive logicprogramming, support vector machines, clustering, Bayesian networks,reinforcement learning, representation learning, similarity and metriclearning, sparse dictionary learning, genetic algorithms, rule-basedmachine learning, and/or learning classifier systems.

Herein, a “machine learning-based model” is a model trained usingmachine learning methods. For brevity's sake, at times, a “machinelearning-based model” may simply be called a “model”. Referring to amodel as being “machine learning-based” is intended to indicate that themodel is trained using machine learning methods (otherwise, “model” mayalso refer to a model generated by methods other than machine learning).

In some embodiments, which involve utilizing a machine learning-basedmodel, a computer is configured to detect the physiological response bygenerating feature values based on the thermal measurements (andpossibly other values), and/or based on values derived therefrom (e.g.,statistics of the measurements). The computer then utilizes the machinelearning-based model to calculate, based on the feature values, a valuethat is indicative of whether, and/or to what extent, the user isexperiencing (and/or is about to experience) the physiological response.Optionally, calculating said value is considered “detecting thephysiological response”. Optionally, the value calculated by thecomputer is indicative of the probability that the user has/had thephysiological response.

Herein, feature values may be considered input to a computer thatutilizes a model to perform the calculation of a value, such as thevalue indicative of the extent of the physiological response mentionedabove. It is to be noted that the terms “feature” and “feature value”may be used interchangeably when the context of their use is clear.However, a “feature” typically refers to a certain type of value, andrepresents a property, while “feature value” is the value of theproperty with a certain instance (sample). For example, a feature may betemperature at a certain ROI, while the feature value corresponding tothat feature may be 36.9° C. in one instance and 37.3° C. in anotherinstance.

In some embodiments, a machine learning-based model used to detect aphysiological response is trained based on data that includes samples.Each sample includes feature values and a label. The feature values mayinclude various types of values. At least some of the feature values ofa sample are generated based on measurements of a user taken during acertain period of time (e.g., thermal measurements taken during thecertain period of time). Optionally, some of the feature values may bebased on various other sources of information described herein. Thelabel is indicative of a physiological response of the usercorresponding to the certain period of time. Optionally, the label maybe indicative of whether the physiological response occurred during thecertain period and/or the extent of the physiological response duringthe certain period. Additionally or alternatively, the label may beindicative of how long the physiological response lasted. Labels ofsamples may be generated using various approaches, such as self-reportby users, annotation by experts that analyze the training data,automatic annotation by a computer that analyzes the training dataand/or analyzes additional data related to the training data, and/orutilizing additional sensors that provide data useful for generating thelabels. It is to be noted that herein when it is stated that a model istrained based on certain measurements (e.g., “a model trained based onTH_(ROI) taken on different days”), it means that the model was trainedon samples comprising feature values generated based on the certainmeasurements and labels corresponding to the certain measurements.Optionally, a label corresponding to a measurement is indicative of thephysiological response at the time the measurement was taken.

Various types of feature values may be generated based on thermalmeasurements. In one example, some feature values are indicative oftemperatures at certain ROIs. In another example, other feature valuesmay represent a temperature change at certain ROIs. The temperaturechanges may be with respect to a certain time and/or with respect to adifferent ROI. In order to better detect physiological responses thattake some time to manifest, in some embodiments, some feature values maydescribe temperatures (or temperature changes) at a certain ROI atdifferent points of time. Optionally, these feature values may includevarious functions and/or statistics of the thermal measurements such asminimum/maximum measurement values and/or average values during certainwindows of time.

It is to be noted that when it is stated that feature values aregenerated based on data comprising multiple sources, it means that foreach source, there is at least one feature value that is generated basedon that source (and possibly other data). For example, stating thatfeature values are generated from thermal measurements of first andsecond ROIs (TH_(ROI1) and TH_(ROI2), respectively) means that thefeature values may include a first feature value generated based onTH_(ROI1) and a second feature value generated based on TH_(ROI2).Optionally, a sample is considered generated based on measurements of auser (e.g., measurements comprising TH_(ROI1) and TH_(ROI2)) when itincludes feature values generated based on the measurements of the user.

In addition to feature values that are generated based on thermalmeasurements, in some embodiments, at least some feature values utilizedby a computer (e.g., to detect a physiological response or train a mode)may be generated based on additional sources of data that may affecttemperatures measured at various facial ROIs. Some examples of theadditional sources include: (i) measurements of the environment such astemperature, humidity level, noise level, elevation, air quality, a windspeed, precipitation, and infrared radiation; (ii) contextualinformation such as the time of day (e.g., to account for effects of thecircadian rhythm), day of month (e.g., to account for effects of thelunar rhythm), day in the year (e.g., to account for seasonal effects),and/or stage in a menstrual cycle; (iii) information about the userbeing measured such as sex, age, weight, height, and/or body build.Alternatively or additionally, at least some feature values may begenerated based on physiological signals of the user obtained by sensorsthat are not thermal cameras, such as a visible-light camera, aphotoplethysmogram (PPG) sensor, an electrocardiogram (ECG) sensor, anelectroencephalography (EEG) sensor, a galvanic skin response (GSR)sensor, or a thermistor.

The machine learning-based model used to detect a physiological responsemay be trained, in some embodiments, based on data collected inday-to-day, real world scenarios. As such, the data may be collected atdifferent times of the day, while users perform various activities, andin various environmental conditions. Utilizing such diverse trainingdata may enable a trained model to be more resilient to the variouseffects different conditions can have on the values of thermalmeasurements, and consequently, be able to achieve better detection ofthe physiological response in real world day-to-day scenarios.

Since real world day-to-day conditions are not the same all the time,sometimes detection of the physiological response may be hampered bywhat is referred to herein as “confounding factors”. A confoundingfactor can be a cause of warming and/or cooling of certain regions ofthe face, which is unrelated to a physiological response being detected,and as such, may reduce the accuracy of the detection of thephysiological response. Some examples of confounding factors include:(i) environmental phenomena such as direct sunlight, air conditioning,and/or wind; (ii) things that are on the user's face, which are nottypically there and/or do not characterize the faces of most users(e.g., cosmetics, ointments, sweat, hair, facial hair, skin blemishes,acne, inflammation, piercings, body paint, and food leftovers); (iii)physical activity that may affect the user's heart rate, bloodcirculation, and/or blood distribution (e.g., walking, running, jumping,and/or bending over); (iv) consumption of substances to which the bodyhas a physiological response that may involve changes to temperatures atvarious facial ROIs, such as various medications, alcohol, caffeine,tobacco, and/or certain types of food; and/or (v) disruptive facialmovements (e.g., frowning, talking, eating, drinking, sneezing, andcoughing).

Occurrences of confounding factors may not always be easily identifiedin thermal measurements. Thus, in some embodiments, systems mayincorporate measures designed to accommodate for the confoundingfactors. In some embodiments, these measures may involve generatingfeature values that are based on additional sensors, other than thethermal cameras. In some embodiments, these measures may involverefraining from detecting the physiological response, which should beinterpreted as refraining from providing an indication that the user hasthe physiological response. For example, if an occurrence of a certainconfounding factor is identified, such as strong directional sunlightthat heats one side of the face, the system may refrain from detectingthat the user had a stroke. In this example, the user may not be alertedeven though a temperature difference between symmetric ROIs on bothsides of the face reaches a threshold that, under other circumstances,would warrant alerting the user.

Training data used to train a model for detecting a physiologicalresponse may include, in some embodiments, a diverse set of samplescorresponding to various conditions, some of which involve occurrence ofconfounding factors (when there is no physiological response and/or whenthere is a physiological response). Having samples in which aconfounding factor occurs (e.g., the user is in direct sunlight ortouches the face) can lead to a model that is less susceptible towrongfully detect the physiological response (which may be considered anoccurrence of a false positive) in real world situations.

After a model is trained, the model may be provided for use by a systemthat detects the physiological response. Providing the model may involveperforming different operations, such as forwarding the model to thesystem via a computer network and/or a shared computer storage medium,storing the model in a location from which the system can retrieve themodel (such as a database and/or cloud-based storage), and/or notifyingthe system regarding the existence of the model and/or regarding anupdate to the model.

A model for detecting a physiological response may include differenttypes of parameters. Following are some examples of variouspossibilities for the model and the type of calculations that may beaccordingly performed by a computer in order to detect the physiologicalresponse: (a) the model comprises parameters of a decision tree.Optionally, the computer simulates a traversal along a path in thedecision tree, determining which branches to take based on the featurevalues. A value indicative of the physiological response may be obtainedat the leaf node and/or based on calculations involving values on nodesand/or edges along the path: (b) the model comprises parameters of aregression model (e.g., regression coefficients in a linear regressionmodel or a logistic regression model). Optionally, the computermultiplies the feature values (which may be considered a regressor) withthe parameters of the regression model in order to obtain the valueindicative of the physiological response; and/or (c) the model comprisesparameters of a neural network. For example, the parameters may includevalues defining at least the following: (i) an interconnection patternbetween different layers of neurons, (ii) weights of theinterconnections, and (iii) activation functions that convert eachneuron's weighted input to its output activation. Optionally, thecomputer provides the feature values as inputs to the neural network,computes the values of the various activation functions and propagatesvalues between layers, and obtains an output from the network, which isthe value indicative of the physiological response.

A user interface (UI) may be utilized, in some embodiments, to notifythe user and/or some other entity, such as a caregiver, about thephysiological response and/or present an alert responsive to anindication that the extent of the physiological response reaches athreshold. The UI may include a screen to display the notificationand/or alert, a speaker to play an audio notification, a tactile UI,and/or a vibrating UI. In some embodiments, “alerting” about aphysiological response of a user refers to informing about one or moreof the following: the occurrence of a physiological response that theuser does not usually have (e.g., a stroke, intoxication, and/ordehydration), an imminent physiological response (e.g., an allergicreaction, an epilepsy attack, and/or a migraine), and an extent of thephysiological response reaching a threshold (e.g., stress and/or angerreaching a predetermined level).

Many physiological responses are manifested in the temperature that ismeasured on various regions of the human face. For example, temperaturesof the nose may be indicative of whether the person is having anallergic reaction. Even though monitoring and analyzing facialtemperatures can be useful for many health-related and lifelogging-related applications, collecting such data over time, whenpeople are going about their daily activities, can be very difficult.Typically, collection of such data involves utilizing thermal camerasthat are bulky, expensive, and need to be continually pointed at aperson's face. Additionally, due to the movements involved in day-to-dayactivities, various image analysis procedures need to be performed, suchas face tracking and image registration, in order to collect therequired measurements. Therefore, there is a need for way to be able tocollect measurements temperatures of the nose, and possibly otherregions of the face, and to utilize them for various applications suchas detecting an allergic reaction. Preferably, the measurements need tobe able to be collected over a long period, while the person performsvarious day-to-day activities.

One application for which thermal measurements of the face may be usefulis to detect an allergic reaction. In one embodiment, a systemconfigured to detect an allergic reaction of a user includes at least aCAM that takes thermal measurements of a region on the nose (TH_(N)) ofthe user, and a computer that detects an allergic reaction of the userbased on TH_(N). Optionally, an allergen may be any substance thatcauses the user to experience an allergic reaction due to the exposureof the user to the allergen (e.g., by consuming, inhaling, and/or cominginto physical contact with the allergen). For example, an allergicreaction may be a reaction to a drug, peanuts, eggs, wheat, dairyproducts, seafood, pollen, dust, and/or perfume.

In one embodiment, CAM is physically coupled to a frame worn on theuser's head (e.g., a frame of glasses or an augmented reality display).Optionally, CAM is located less than 15 cm from the user's face.Optionally, CAM weighs less than 10 g, 5 g or 1 g. Optionally, CAM usesa thermopile, a pyroelectric sensor, or a microbolometer sensor, whichmay be a focal-plane array sensor. For example, CAM may be the thermalcameras 48 and/or 49, which are illustrated in FIG. 1b , or the thermalcamera 540 illustrated in FIG. 22.

Optionally, multiple CAMs may be utilized to obtain measurements ofvarious ROIs such as different regions/sides of the nose, mouth and/orcheeks. For example, allergic reaction may cause red eyes, itchy eyes,tearing eyes, swollen eyelids, and/or burning eyes/eyelids. In somecases, a thermal camera that captures a region on the periorbital(TH_(peri)) around at least one of the eyes may detect an eye allergysymptom before the user is aware of the allergic reaction and/or used toassess the extent of the allergic reaction. As another example, allergicreaction may cause hives (urticaria) around the mouth and/or other partsof the face. In some cases, a thermal camera that captures the areaaround the mouth (TH_(lips)) may detect the hives around the mouthbefore the user is aware of the allergic reaction and/or used to assessthe extent of the allergic reaction. In still some cases, thermalmeasurements of regions on the right and/or left cheeks (TH_(ch)) mayhelp detecting the allergic reaction.

The computer is configured, in one embodiment, to detect the allergicreaction based on TH_(N) and optionally other data, such as TH_(CH),TH_(peri), and/or TH_(lips) mentioned above and/or other sources ofinformation mentioned below. In one embodiment, detecting the allergicreaction may involve one or more of the following: determining whetherthe user is experiencing an allergic reaction, and/or determining theextent of the allergic reaction. Optionally, the extent of the allergicreaction may be indicative of the severity of the allergic reaction,and/or the duration of the allergic reaction (e.g., total time of theallergic reaction and/or the time remaining until the allergic reactionsubsides).

In some cases, changes to temperatures at regions of the face (e.g., inthe nasal area) occur quickly at the initial stages of an allergicreaction. Thus, the computer may detect the allergic reaction at itsinitial stages even before the user is aware of the allergic reaction.Thus, in some embodiments, detecting the allergic reaction involvesdetecting an onset of the allergic reaction, which may involvedetermining the time until the reaction reaches its peak severity (e.g.,a rash, coughing, respiratory distress, sneezing) and/or determining theexpected degree of severity (extent) of the allergic reaction.

In some cases, at the time the allergic reaction is identified, a userhaving the allergic reaction may not be aware of the allergic reaction.e.g., because the symptoms are not strong enough at the time. Thus,being notified about an allergic reaction before its full manifestationmay have an advantage, in some embodiments, of allowing the user to takeearly action to alleviate and/or decrease the symptoms (e.g., takeantihistamines) or seek medical attention.

In some allergic reactions, the nasal temperature can rise rapidlywithin minutes, before other more noticeable symptoms may manifestthemselves (e.g., sneezing, itching, and/or respiratory problems). Thus,rising nasal temperatures may serve as an indication of an allergicreaction. For example, a fast increase due to an allergic reaction maycorrespond to an increase of more than 0.8° C. within a period of lessthan 10 minutes, or even less than 5 minutes.

FIG. 21a and FIG. 21b illustrate a scenario in which a user is alertedabout an expected allergic reaction. In FIG. 21a , the user's nasaltemperature is normal. At that time, a cat, to which the user isallergic, walks past the user. FIG. 21b illustrates the situationshortly after. The user's nasal temperature has increased, and based onthermal measurements of the nasal region, a computer issues an alert tothe user about the expected allergic reaction. Note that at the time thealert is issued, the user may not be aware of any symptoms of theallergic reaction. Receiving an early warning in this case may enablethe user to take measures to alleviate the effects of the allergicreaction, such as taking an antihistamine medicine.

There are various ways the computer may utilize TH_(N) and possiblyother thermal measurements such as TH_(CH), TH_(peri), and/or TH_(lips),in order to detect the allergic reaction. In one embodiment, thecomputer may compare values derived from TH_(N) (and/or from TH_(CH),TH_(peri), and/or TH_(lips)) to a certain threshold, and determinewhether the threshold is reached (which is indicative of an occurrenceof the allergic reaction). Optionally, the threshold is determined basedon previous thermal measurements of the user. Optionally, the thresholdis determined based on previous thermal measurements of other users. Inanother embodiment, the computer may determine a similarity between areference time series corresponding to the allergic reaction and TH_(N)and optionally the other thermal measurements (or a time series derivedtherefrom). Optionally, when a sufficiently high similarity is detected,the computer may interpret that as an indication of an occurrence of theallergic reaction. The reference time series may be generated based onprevious thermal measurements of the user and/or of other users.

In yet another embodiment, the computer may generate feature valuesbased on thermal measurements comprising TH_(N) and optionally TH_(CH),TH_(peri), and/or TH_(lips), and utilize a machine learning-based modelto calculate, based on the feature values, a value indicative of whetherthe allergic reaction occurred and/or indicative of an extent of theallergic reaction (calculating the value be considered herein as“detecting the allergic reaction”). Optionally, the model was trainedbased on previous thermal measurements of the user. For example, theprevious thermal measurements may include a first set of thermalmeasurements taken while the user had an allergic reaction, and a secondset of thermal measurements taken while the user did not have anallergic reaction. In this example, the model may be considered apersonalized model for the user. Additionally or alternatively, themodel may be trained on thermal measurements of other users (e.g., ageneral model). Optionally, different models may be created to detectdifferent types of allergic reactions, to detect allergic reactions todifferent allergens, and/or to detect different extents of an allergicreaction.

In one example, detection of the allergic reaction may involve thecomputer performing the following: (i) generating feature values basedon thermal measurements comprising TH_(N) and optionally TH_(CH),TH_(peri), and/or TH_(lips); and (ii) utilizing a model to detect theallergic reaction based on the feature values. Optionally, the model wastrained based on previous thermal measurements of the user comprisingTH_(N) and optionally TH_(CH), TH_(peri), and/or TH_(lips), which weretaken while the user had an allergic reaction. Alternatively, the modelwas trained based on a first set of previous thermal measurements of theuser comprising TH_(N) and optionally TH_(CH), TH_(peri), and/orTH_(lips), which were taken while the user had an allergic reaction, anda second set of previous thermal measurements of the user comprisingTH_(N) and optionally TH_(CH), TH_(peri), and/or TH_(lips), which weretaken while the user did not have an allergic reaction.

In some embodiments, detecting the allergic reaction may involveutilizing baseline TH_(N), most of which were taken when the user didnot have an allergic reaction. Thus, detecting the allergic reaction mayrely on observing a change relative to typical temperatures at the ROIs.In one example, the computer detects the allergic reaction based adifference between TH_(N) and a baseline value determined based on a setof previous TH_(N) taken with CAM. In this example, most of TH_(N)belonging to the set were taken while the user had an allergic reaction,or within thirty minutes before or after the user had an allergicreaction.

Confounding factors such as extensive physical activity, touching thenose, and/or direct sunlight aimed at the nose may lead, in someembodiments, to less accurate detections of an allergic reaction (e.g.,by increasing the frequency of false detections of the allergicreaction). In some embodiments, the system may include a sensor thattakes additional measurements (m_(conf)) of the user, and/or of theenvironment in which the user was in while TH_(N) were taken.Optionally, m_(conf) are indicative of an extent to which a confoundingfactor occurred while TH_(N) were taken. Another approach that may beutilized by the computer is to generate feature values based on m_(conf)and to utilize these feature values in the detection of the allergicreaction.

Some of the embodiments described herein may be utilized to identifypotential causes for the change (e.g., rise) of the temperature at anROI. These causes may include inhaled allergens, food, drugs, and/orvarious chemicals which the user might have been exposed to (e.g., viaingestion, inhalation, and/or physical contact). In one embodiment, thecomputer may identify a potential allergen substance by estimating atime of exposure to the allergen from data indicative of a deviationover time of mean nasal temperature from a baseline and identifying thesubstances consumed by the user, and/or to which the user was exposed,around that time. For example, by identifying based on TH_(N) when thenasal temperature started to rise, and taking into account the timerequired for the allergic reaction to be manifested via a temperaturerise, a window of time can be determined during which the user waslikely exposed to the allergen. Examining which substances the user wasexposed to during the window can yield a list of one or more potentialallergen substances. Optionally, the system alerts the user about theone or more potential allergen substances. Optionally, the system storesin a database potential allergen substances identified based on dataindicative of a deviation over time of mean nasal temperature frombaseline (such as allergens identified based on deviation over time ofmean nasal temperature from baseline). In some embodiments, the systemincludes a camera that captures images of substances consumed by theuser. Optionally, the camera is mounted to a frame worn on the user'shead. Optionally, the system displays to the user an image of asubstance associated with the potential allergen substance.

There are various known systems that may be utilized to monitor whatsubstances a user was exposed to and/or what substances a user consumed.For example, systems that may be utilized to determine what the user ateor drank are described in the patent application US 20110318717(Personalized Food Identification and Nutrition Guidance System), in theU.S. Pat. No. 9,053,483 (Personal audio/visual system providing allergyawareness), and in the U.S. Pat. No. 9,189,021 (Wearable food nutritionfeedback system). Additionally, obtaining indications of possibleallergens to which the user was exposed is described in the U.S. Pat.No. 9,000,933 (Automated allergy alerts). In one embodiment, uponidentifying an increase in nasal temperature, the system can identifythe potential cause to be one of the substances to which the user wasexposed during a predetermined preceding duration, such as the preceding20 min, 10 min, or 5 min.

FIG. 22 illustrates how the system may be utilized to identify a triggerof an allergic reaction. CAM 540 is coupled to a frame of eyeglassesworn by the user and takes thermal measurements of a region on theuser's nose 541, while the user eats different types of food. The dottedlines on the graph indicate when the user started eating each type offood. The nasal temperature increases shortly after starting eating thepersimmon; however, it may reach a threshold indicating an allergicreaction only after some time, during which the user eats the pizza orthe ice cream. Thus, in this case, the allergic reaction should likelybe attributed to the persimmon or the soup, and not attributed to thepizza or the ice cream. Optionally, outward-facing head-mountedvisible-light camera 542 takes images of the food the user eats, and thecomputer uses image processing to detect the types of food.

Another approach for identifying a cause of an allergic reaction (a“trigger” of an allergic reaction), involves analysis of potentialtriggers and the user's detected response when affected by the potentialtriggers. In one embodiment, the computer is further configured to:receive indications of times during which the user was exposed topotential triggers of the allergic reaction, and select a trigger, fromamong the potential triggers, based on the indications and extents ofthe allergic reaction detected based on TH_(N). Optionally, during mostof the time the user was affected by the trigger, an effect of thetrigger, as manifested via changes to TH_(N), was higher than effects ofmost of the potential triggers. Optionally, a camera is utilized to takeimages of the surroundings of the user, and the computer generates atleast some of the indications based on analysis of the images. In oneexample, the exposure to the potential triggers involves consuming acertain drug and/or a certain food item. In another example, theexposure to the potential triggers involves being exposed to pollen,dust, and/or a certain cosmetics product. In still another example, theexposure to the potential triggers involves the user being at a certainlocation, and/or the user being in contact with a certain animal.

Due to the mostly symmetric nature of the human body, when the faceundergoes temperature changes, e.g., due to external factors such as thetemperature in the environment or internal factors such as anactivity-related rise in body temperature, the changes to the face aregenerally symmetric. That is, the temperature changes at a region ofinterest (ROI) on the left side of the face (e.g., the left side of theforehead) are similar to the temperature changes at the symmetric ROT onthe right side of the face (e.g., the right side of the forehead).However, when the temperature on the face changes in an asymmetric way,this can be indicative of various physiological responses and/orundesirable phenomena. Some examples of phenomena that may be identifiedby detecting asymmetric thermal patterns (“thermal asymmetry”) on auser's face include a headache, sinusitis, nerve damage, some types ofstrokes, orofacial pain, and Bell's palsy. Additionally, some forms ofdisorders such as Attention Deficit Hyperactivity Disorder (ADHD),stress, anxiety, and/or depression can also be identified based onthermal asymmetry of the forehead, and in some cases of other regions ofthe face.

In other cases, and sometime depending on personal characteristics ofthe user, certain physiological responses may manifest differently ondifferent sides of the face. In particular, the temperatures atdifferent positions on the right side of the face may not be a mirrorimage of the temperatures at the corresponding positions on the leftside of the face. Thus, having two or more thermal cameras pointed atdifferent areas of the face can, in some embodiments, help make moreaccurate detections of a physiological response. For example, stress maybe manifested with some people by the cooling of an area on one side ofthe nose more than the symmetric area on the other side. Similarly, withsome people, an allergic reaction may manifest by the nose heating todifferent extents on each of its sides. Thus, having, in this example,two or more thermal cameras pointed at different sides of the nose, mayenable a more accurate detection of the physiological response.

Measuring and utilizing the asymmetric data also improves the robustnessof the system against interferences that may cause an asymmetric thermaleffect, such as an external heat source located to the user's side, acooling air-conditioner that blows air from the top, touching and/orwiping one side of the face, and for some people also eating and/orconducting a physical activity. Therefore, utilizing thermal cameraspointed at symmetric ROIs may improve the system's ability to detect aphysiological response compared to the case in which just one thermalcamera is used.

In one embodiment, a system configured to collect thermal measurementsindicative of thermal asymmetry on a user's face includes first andsecond inward-facing head-mounted thermal cameras (CAM1 and CAM2).Optionally, CAM1 and CAM2 are physically coupled to a frame worn on theuser's head, and are located less than 15 cm, 5 cm, or 2 cm from theuser's face. Optionally, CAM1 and CAM2 are located at least 0.5 cm tothe right and to the left of the vertical symmetry axis that divides theface, respectively. Optionally, each of CAM1 and CAM2 weighs below 10 g,5 g, or 1 g.

CAM1 and CAM2 take thermal measurements of regions on the right and leftsides of the face (TH_(ROI1) and TH_(ROI2), respectively) of the user,and optionally do not occlude ROI₁ and ROI₂. Optionally, CAM1 and CAM2are based on thermopile, microbolometer, or pyroelectric sensors, whichmay be focal-plane array sensors. Optionally, ROI₁ and ROI₂ havesymmetric overlapping above 60%. In one example, CAM1 and CAM2 may bethermal cameras 120 and 122 in FIG. 10. In another example, CAM and CAM2are thermal cameras 126 and 128 in FIG. 11.

The symmetric overlapping is considered with respect to the verticalsymmetry axis that divides the face to the right and left portions. Thesymmetric overlapping between ROI₁ and ROI₂ may be observed by comparingthe overlap between ROI₁ and a mirror image of ROI₂, where the mirrorimage is with respect to a mirror that is perpendicular to the front ofthe face and whose intersection with the face is along the verticalsymmetry axis (which goes through the middle of the forehead and themiddle of the nose). Depending on the application for which the thermalmeasurements are utilized, the ROIs may have different degrees ofsymmetric overlapping. In one example, the symmetric overlapping betweenROI₁ and ROI₂ is above 80% of the smallest area from among the areas ofROI₁ and ROI₂. In another example, the overlap between ROI₁ and ROI₂ isabove 25% and below 80% of the smallest area from among the areas ofROI₁ and ROI₂.

Depending on the locations of ROI₁ and ROI₂, in different embodiments,CAM1 and CAM2 may be located in specific locations on the frame and/orwith respect to the face. In one example, ROI₁ and ROI₂ are on the noseand/or a region on the mouth, and CAM1 and CAM2 are located outside theexhale streams of the mouth and/or nostrils.

In one embodiment, each of CAM1 and CAM2 is located less than 10 cm fromthe face and there are angles greater than 20° between the Frankforthorizontal plane and the optical axes of CAM1 and CAM2.

Due to the angle between the optical axis of CAM1 and CAM2 and theFrankfort horizontal plane, in some embodiments, the Scheimpflugprinciple, may be employed in order to capture sharper images. Forexample, when the user wears the frame, CAM1 and/or CAM2 may have acertain tilt greater than 2° between their sensor and lens planes, inorder to produce the sharper images.

In one embodiment, CAM1 and CAM2 utilize focal-plane array (FPA)sensors. Optionally, each FPA includes at least 6 or at least 12 sensingelements (pixels). Optionally, there are angles greater than 20° betweenthe Frankfort horizontal plane and the optical axes of CAM1 and CAM2.Optionally. CAM1 is located to the right of the vertical symmetry axisand takes thermal measurements of a first region of interest(TH_(ROI1)), where ROI₁ covers more of the right side of the face thanof the left side of the face; CAM2 is located to the left of thevertical symmetry axis and takes thermal measurements of a second regionof interest (TH_(ROI2)), where ROI₂ covers more of the left side of theface than of the right side of the face. Optionally, the cameras do notocclude ROI₁ and ROI₂. Alternatively, the cameras occlude at least partof ROI₁ and ROI₂.

In some embodiments, the system for collecting thermal measurementsindicative of thermal asymmetry on a user's face includes a computer.Optionally, the computer detects a physiological response based on thethermal measurements.

In one embodiment, the detection of the physiological response utilizesa personalized model of the user. Optionally, the computer (i) generatesfeature values based on TH_(ROI1) and TH_(ROI2), and (ii) utilizes amodel to detect the physiological response based on the feature values.Optionally, at least some feature values used to detect thephysiological response may be generated based on additional sources ofinformation (other than CAM1 and CAM2), such as additional thermalcameras, additional sensors that measure physiological signals of theuser (e.g., heart rate or galvanic skin response), and/or additionalsensors that measure the environment. Optionally, the model is trainedbased on previous TH_(ROI1) and TH_(ROI2) taken while the user had thephysiological response. Optionally, the physiological response involvesthe user experiencing stress, mental workload, fear, sexual arousal,anxiety, pain, a headache, dehydration, intoxication, and/or a stroke.Optionally, the physiological response is associated with facial thermalasymmetry, and the model was trained based on previous feature valuestaken during different days. Optionally, the previous feature valuesinclude; a first set of feature values generated based on TH_(ROI1) andTH_(ROI2) taken while the user had the physiological response, and asecond set of feature values generated based on TH_(ROI1) and TH_(ROI2)taken while the user did not have the physiological response.

In different embodiments, the difference between TH_(ROI1) and TH_(ROI2)may be interpreted in different ways. In one embodiment, an extent of aphysiological response may be proportional to the difference betweenTH_(ROI1) and TH_(ROI2) when the value of the difference is in a certainrange. Optionally, when the value of the difference is outside of therange, this may be indicative of the occurrence of other phenomena(which are not the physiological response). In another embodiment, whenthe value of the difference between TH_(ROI1) and TH_(ROI2) reaches athreshold, that is indicative of an occurrence of the physiologicalresponse. In yet another embodiment, at least one feature value utilizedby a predictor that predicts occurrences of the physiological responseis based on the value of the difference between TH_(ROI1) and TH_(ROI2).

Often a change in the thermal asymmetry may be indicative of aphysiological response. Optionally, the computer detects a change tothermal asymmetry on the face based on a change between thermalmeasurements taken at different times. The computer may furthercalculate the extent of the physiological response based on the change.This calculation can be performed in different ways, as described below.

Additional CAMs may be utilized to take thermal measurements used fordetecting the physiological response. FIG. 9 illustrates one embodimentof a system that collects thermal measurements indicative of thermalasymmetry on a user's face, which involves additional CAMs. The systemincludes a frame 90, which has six CAMs coupled to it (some embedded inprotruding arms). CAMs 91 and 92 are located on arms on the right andleft sides of the top of the frame 90, respectively, and take thermalmeasurements of regions on the right and left sides of the forehead (97and 98, respectively). CAMs 93 and 94 are located on the right and leftsides of the frame 90 near the nose, respectively, and take thermalmeasurements of regions on the right and left periorbital areas (99 and100), respectively. CAMs 95 and 96 are located on arms connected to thebottom of right and left rims, respectively, and take thermalmeasurements of right and left lower regions of the face (101 and 102,respectively). Optionally, some (or all) of the cameras contain multiplesensing elements.

In one embodiment, the system for collecting thermal measurementsindicative of thermal asymmetry on a user's face further includes thirdand fourth CAMs (in addition to CAM1 and CAM2), each of which: weighsbelow 10 g, is physically coupled to the frame, and is located less than15 cm from the face. The third and fourth CAMs take thermal measurementsof regions on the right and left sides of the upper lip (TH_(ROI3) andTH_(ROI4), respectively) of the user, without occluding the upper lip.Optionally, the symmetric overlapping between the regions on the rightand left sides of the upper lip is above 60%. Optionally, the systemincludes a computer that (i) generates feature values based onTH_(ROI1), TH_(ROI2), TH_(ROI3), and TH_(ROI4), and (ii) utilizes amodel to detect a physiological response based on the feature values.Optionally, the model was trained based on previous TH_(ROI), TH_(ROI2),TH_(ROI3), and TH_(ROI4) taken while the user had a physiologicalresponse associated with at least one of the following: stress, mentalworkload, fear, sexual arousal, anxiety, pain, a headache, dehydration,intoxication, and a stroke.

In another embodiment, ROI₁ and ROI₂ are on the right and left sides ofthe forehead, respectively, and the system further includes at leastthird and fourth CAMs, located less than 10 cm from the face, which takethermal measurements of regions on the right and left periorbital areas(TH_(ROI3) and TH_(ROI4), respectively). Optionally, the system includesa computer that utilizes a model to detect an emotional state and/orstress level based on TH_(ROI1), TH_(ROI2), TH_(ROI3), and TH_(ROI4).Optionally, the model was trained based on previous TH_(ROI1),TH_(ROI2), TH_(ROI3), and TH_(ROI4) taken during different days.Optionally, the system includes additional fifth and sixth CAMs, locatedless than 10 cm from the face, which take thermal measurements ofregions on the right and left cheeks (TH_(ROI5) and TH_(ROI6),respectively). Optionally, the computer detects the physiologicalresponse also based on TH_(ROI5) and TH_(ROI6) (e.g., by generatingbased on TH_(ROI5) and TH_(ROI6) at least some of the feature valuesused to detect the physiological response).

In yet another embodiment, the system further includes third and fourthCAMs for taking thermal measurements of the environment to the right andto the left of the face (TH_(ENV1) and TH_(ENV2), respectively). Thecomputer utilizes TH_(ENV1) and TH_(ENV2) to identify asymmetryresulting from the environment rather than from a physiologicalresponse. For example, the computer may generate feature values based onTH_(ENV1) and TH_(ENV2), and utilize these feature values, in additionto feature values generated based on thermal measurements of the ROIs onthe face, in order to detect the physiological response. Optionally, thethird and fourth CAMs are based on at least one of the following sensortypes: a thermopile, a pyroelectric sensor, and a microbolometer.Optionally, the environmental cause of the asymmetry involves at leastone of the following: sunlight, air blowing from an air-conditioner,radiation from a heater, and radiation from an oven.

The following examples of physiological responses may be identifiedutilizing embodiments of the system for collecting thermal measurementsindicative of thermal asymmetry on a user's face.

There are various forms of sinusitis that may be detected utilizingdifferent embodiments of the system. In one embodiment, ROI₁ and ROI₂are on the right and left anterior sinuses, respectively. Optionally,the computer utilizes a model to detect sinusitis based on TH_(ROI1) andTH_(ROI2) (as described above). Optionally, the data used to train themodel includes TH_(ROI1) and TH_(ROI2) taken from other users who sufferfrom maxillary sinusitis, frontal sinusitis, unilateral frontalsinusitis, and/or unilateral maxillary sinusitis. In a first example,ROI₁ and ROI₂ are on the right and left anterior sinus group,respectively. Optionally, the right/left anterior sinus group includesthe right/left frontal sinus, the right/left maxillary sinus, and theright/left anterior ethmoid sinus. In a second example, ROI₁ and ROI₂are on the user's right and left frontal sinuses, respectively, and thecomputer detects an occurrence of a unilateral frontal sinusitis. In athird example, ROI₁ and ROI₂ are on the user's right and left maxillarysinuses, respectively, and the computer detects an occurrence of aunilateral maxillary sinusitis.

Some forms of strokes may be detected using embodiments of the system.In a first example, ROI₁ and ROI₂ are on the right and left superficialtemporal arteries. In a second example, each of ROI₁ and ROI₂ coverabove 20%, or above 40%, of the right and left sides of the face thatinclude exposed facial skin between the mouth level and the eyebrowlevel, respectively (e.g., the right and left cheeks and/or the rightand left sides of the upper lip). Herein, “exposed facial skin” refersto facial skin that does not have excessive hair growth, such as a beardthat usually damages the ability of CAM to measure the skin under thebeard. In these two examples, a computer may detect whether the user hasa stroke based on changes observed by comparing TH_(ROI1) and TH_(ROI2)taken from the user during different days. Optionally, if theprobability that the user has a stroke reaches a certain threshold, suchas at least 5%, 25%, or 50%, then the user and/or a third party arealerted about this finding so the user can receive immediate medicalattention.

FIG. 19 illustrates a scenario in which an alert regarding a possiblestroke is issued. The figure illustrates a user wearing a frame with atleast two CAMs (562 and 563) for measuring ROIs on the right and leftcheeks (ROIs 560 and 561, respectively). The measurements indicate thatthe left side of the face is colder than the right side of the face.Based on these measurements, and possibly additional data, the systemdetects the stroke and issues an alert. Optionally, the user's facialexpression is slightly distorted and asymmetric, and a VCAM providesadditional data in the form of images that may help detecting thestroke.

Various forms of nerve damage often cause detectable thermal differenceson the face. At times, the thermal differences may manifest prior tochanges to the appearance of the face. Thus, thermal measurements may beutilized for early detection of nerve damage, which may improve theoutcome of a treatment. For example, in one embodiment, ROI₁ and ROI₂may each be on the periorbital area around the eyes, the nose, and/orthe mouth. Optionally, the computer may identify nerve damage based onchanges observed by comparing TH_(ROI1) and TH_(ROI2) taken from theuser during different days, and/or by using a model trained based onmeasurements of other users taken while they had nerve damages.

Headaches (which also include migraines), symptomatic behavior ofAttention Deficit Hyperactivity Disorder (ADHD), and/or anger attacksare physiological responses that may also be detected by embodimentsdescribed herein. In one embodiment, detecting these physiologicalresponses is done with a system in which ROI₁ and ROI₂ are on the rightand left sides of the user's forehead. Alternatively, ROI₁ and ROI₂ maycover right and left regions on the periorbital areas, the nose, and/orthe mouth. Optionally, the computer detects headaches utilizing a modelthat was trained based on previous TH_(ROI1) and TH_(ROI2) taken duringdifferent days, optionally including samples taken while the user had aheadache and while the user did not have a headache.

Additionally, in some embodiments, a relationship between the stress theuser feels and headache the user has may be studied. Optionally, thecomputer receive training data comprising physiological measurementsindicative of levels of stress of the user, values indicative ofdurations during which the user felt stressed, and values indicative ofdurations during which the user had a headache. The computer utilizes amachine learning-based training algorithm to train the model based onthe training data. The model may be used to detect a headache based onTH_(ROI1) and TH_(ROI2) and optionally, additional values indicative ofstress the user felt.

Orofacial pain often results from dental causes (e.g., toothache causedby pulpitis or a dental abscess). Such pain may also be detectedutilizing some embodiments of the system. In one embodiment, ROI₁ andROI₂ are on the right and left sides of at least one of the jaws.Optionally, the computer detects orofacial pain based on TH_(ROI1) andTH_(ROI2) utilizing a model that was trained based on previous TH_(ROI1)and TH_(ROI2) taken during different days.

Bell's palsy is another medical disorder that may be identified based onthermal measurements. In one embodiment, the system includes a computerthat detects Bell's palsy based on comparing TH_(ROI1) and TH_(ROI2)taken from the user during different days. Optionally, the systemfurther includes a VCAM for taking photos of the face, and the computeranalyzes the photos for asymmetry in order to improve the probability ofidentifying Bell's palsy. For example, the detection of Bell's palsy maybe done based on feature values that include feature values generatedbased on the thermal measurements (e.g., corresponding to differences invalues of thermal measurements at the same locations during differenttimes), and feature values generated based on images taken by VCAM(e.g., corresponding to differences in facial features at the samelocations during different times). Optionally, the system suggests theuser to take a medical examination when the facial thermal asymmetryreaches a threshold for more than a predetermined duration (such as 1minute, 5 minutes, or more than 30 minutes).

Experiencing stress is generally detrimental to people's health.Reducing the amount of stress a user experiences in day-to-day lifetypically requires knowing when the user is stressed, for how long, andin what conditions. While many physiological responses, includingstress, are manifested in the temperatures and/or temperature changes atvarious regions of the human face, collecting such data over time whenpeople are going through their daily activities can be very difficult.Typically, collection of such data involves utilizing thermal camerasthat are bulky, expensive and need to be continually pointed at aperson's face. Additionally, due to the people's movements in theirday-to-day activities, collecting the required measurements ofteninvolves performing various complex image analysis procedures, such asprocedures involving image registration and face tracking. Thus, thereis a need to be able to collect thermal measurements at various regionsof a person's face in order to detect stress. Preferably, themeasurements are to be collected over a long period while the personperforms various day-to-day activities, such as one or more of eating,drinking, talking, moving around, and being in a place where thetemperature and humidity are not strictly controlled.

Some of the disclosed embodiments may be utilized to detect a stresslevel of a user based on thermal measurements of the user's face, suchas the periorbital areas (i.e., areas around the eyes). In oneembodiment, a system configured to detect a stress level includes a CAMand a computer. CAM takes thermal measurements of a region on aperiorbital area (TH_(ROI1)) of the user, and is located less than 10 cmfrom the user's head. The computer detects the stress level based onTH_(ROI1).

In one embodiment, in which the region is on the periorbital area of theright eye, the system further includes a second inward-facinghead-mounted thermal camera (CAM2), which is located less than 10 cmfrom the user's head and takes thermal measurements of a region on theperiorbital area of the left eye (TH_(ROI2)). Optionally, the computerdetects the stress level based on both TH_(ROI1) and TH_(ROI2).Optionally, CAM and CAM2 are located at least 0.5 cm to the right and tothe left of the vertical symmetry axis (which goes through the middle ofthe forehead and the middle of the nose), respectively. Optionally, eachof CAM and CAM2 weighs below 10 g and is based on a thermopile, amicrobolometer, or a pyroelectric sensor, which may be a focal-planearray sensor.

It is to be noted that while various embodiments may utilize a singleCAM, due to asymmetrical placement of blood vessels in the face, thermalemissions of faces of many people are asymmetric to a certain extent.That is, the pattern of thermal emission from the left side of the facemay be different (possibly even noticeably different) from the patternof thermal emission from the right side of the face. Thus, for example,the temperature changes at the periorbital areas, in response toexperiencing at least a certain level of stress, may be asymmetric forsome users. The fact that various embodiments described below mayinclude two (or more) CAMs that take measurements of ROIs coveringdifferent sides of the face (referred to as TH_(ROI1) and TH_(ROI2)) canenable the computer to account for the thermal asymmetry when detectingthe stress level.

In some cases, interferences (such as an external heat source, touchingone of the eyes, or an irritated eye) cause an asymmetric effect on theright and left periorbital areas. As a result, utilizing right and leftCAMs, which are located in different angles relative to the interferingsource, provides the computer additional data that can improve itsperformances. The following are some examples of various ways in whichthe computer may account for the asymmetry when detecting the stresslevel based on TH_(ROI1) on and TH_(ROI2), which include measurements ofthe of regions on the periorbital areas of the right and left eyes ofthe user, respectively.

In one embodiment, when comparing TH_(ROI1) and TH_(ROI2) to thresholds,the computer may utilize different thresholds for TH_(ROI1) andTH_(ROI2), in order to determine whether the user experienced a certainlevel of stress. Optionally, the different thresholds may be learnedbased on previous TH_(ROI1) and TH_(ROI2), which were measured when theuser experienced the certain level of stress and/or suffered fromcertain interferences.

In another embodiment, the computer may utilize different reference timeseries to which TH_(ROI1) and TH_(ROI2) are compared in order todetermine whether the user experienced the certain level of stress.Optionally, accounting for the asymmetric manifestation of the stress isreflected in the fact that a reference time series to which TH_(ROI1) iscompared is different from a reference time series to which TH_(ROI2) iscompared.

In yet another embodiment, when the computer utilizes a model tocalculate a stress level based on feature values generated based onTH_(ROI1) and/or TH_(ROI2). Optionally, the feature values include: (i)at least first and second feature values generated based on TH_(ROI1)and TH_(ROI2), respectively, and/or (ii) a third feature valueindicative of the magnitude of a difference between TH_(ROI1) andTH_(ROI2). In this embodiment, the computer may provide differentresults for first and second events that involve the same average changein TH_(ROI1) and TH_(ROI2), but with different extents of asymmetrybetween TH_(ROI1) and TH_(ROI2), and/or different magnitudes ofinterferences on the right and left eyes.

In still another embodiment, the computer may utilize the fact thatasymmetric temperature changes occur when the user experiences stress inorder to distinguish between stress and other causes of temperaturechanges in the periorbital areas. For example, drinking a hot beverageor having a physical exercise may cause in some people a more symmetricwarming pattern to the periorbital areas than stress. Thus, if such amore symmetric warming pattern is observed, the computer may refrainfrom identifying the temperature changes as being stress-related.However, if the warming pattern is asymmetric and corresponds totemperature changes in the periorbital areas of the user when the userexperiences stress, then the computer may identify the changes in thetemperatures being stress-related.

The computer may employ different approaches when detecting the stresslevel based on TH_(ROI1) (and possibly other sources of data such asTH_(ROI2)). In one embodiment, the computer may compare TH_(ROI1) (andpossibly other data) to a threshold(s), which when reached wouldindicate a certain stress level. In another embodiment, the computer maygenerate feature values based on TH_(ROI1) and TH_(ROI2), and utilize amodel (also referred to as a “machine learning-based model”) tocalculate a value indicative of the stress level based on the featurevalues (calculating the value indicative of the stress level may beconsidered herein as “detecting the stress level”). At least some of thefeature values are generated based on TH_(ROI1). Optionally, at leastsome of the feature values may be generated based on other sources ofdata, such as TH_(ROI2) and/or TH_(ROI3) (described below). Optionally,the model was trained based on samples comprising feature valuesgenerated based on previous TH_(ROI1) (and possibly other data, asexplained below), and corresponding labels indicative of a stress levelof the user. Optionally, the data used to train the model includesprevious TH_(ROI1) taken while the user was under elevated stress, andother previous TH_(ROI1) taken while the user was not under elevatedstress. Optionally, “elevated stress” refers to a stress level thatreaches a certain threshold, where the value of the threshold is setaccording to a predetermined stress scale (examples of stress scales aregiven further below). Optionally, “elevated stress” refers to aphysiological state defined by certain threshold values of physiologicalsignals (e.g., pulse, breathing rate, and/or concentration of cortisolin the blood).

In a first embodiment, when the stress level exceeds a certain value,TH_(ROI1) reach a threshold, and when the stress level does not exceedthe certain value, TH_(ROI1) do not reach the threshold. Optionally, thestress level is proportional to the values of TH_(ROI1) (which arethermal measurements of the region on the periorbital area), such thatthe higher TH_(ROI1) and/or the higher the change to TH_(ROI1) (e.g.,with reference to a baseline), the higher the stress level.

In a second embodiment, the computer detects the stress level based on adifference between TH_(ROI1) and a baseline value determined based on aset of previous measurements taken by CAM. Optionally, most of themeasurements belonging to the set were taken while the user was notunder elevated stress.

In a third embodiment, the stress level is detected using a model andfeature values generated based on additional measurements (m_(conf)) ofthe user and/or of the environment in which the user was in whileTH_(ROI1) were taken, m_(conf) may be taken by sensor 461. Optionally,m_(conf) are indicative of an extent to which a confounding factoroccurred while TH_(ROI1) were taken. The following are some examples ofsources of information for m_(conf), which may be used to detect thestress level.

In a first example, m_(conf) are physiological signals such as a heartrate, heart rate variability, galvanic skin response, a respiratoryrate, and respiratory rate variability, which are taken using sensorssuch as PPG, ECG, EEG, GSR and/or a thermistor.

In a second example, m_(conf) represent an environmental conditionand/or a situation of the user that may be considered a confoundingfactor, such as an indication of whether the user touched at least oneof the eyes, an indication of whether the user is engaged in physicalactivity (and possibly the type and/or extent of the physical activity),temperature, humidity, IR radiation level, and a noise level.Optionally, the one or more values are obtained based on using anaccelerometer, a pedometer, a humidity sensor, a miniature radar (suchas low-power radar operating in the range between 30 GHz and 3,000 GHz),a miniature active electro-optics distance measurement device (such as aminiature Lidar), an anemometer, an acoustic sensor, and/or a lightmeter.

In a third example, m_(conf) represent properties describing the user,such as the user's age, gender, marital status, occupation, educationlevel, health conditions, and/or mental health issues that the user mayhave.

Stress may be thought of as the body's method of reacting to achallenge. Optionally, stress may be considered a physiological reactionto a stressor. Some examples of stressors include mental stressors thatmay include, but are not limited to, disturbing thoughts, discontentwith something, events, situations, individuals, comments, or anything auser may interpret as negative or threatening. Other examples ofstressors include physical stressors that may put strain on the body(e.g., very cold/hot temperatures, injury, chronic illness, or pain). Inone example, a (high) workload may be considered a stressor. The extentto which a user feels stressed is referred to herein as a “stress level”and being under a certain level of stress may be referred to herein as“experiencing a certain stress level”. Depending on the embodiment, astress level may be expressed via various types of values, such as abinary value (the user is “stressed” or “not stressed”, or the user isunder “elevated stress” or “not under elevated stress”), a categorialvalue (e.g., no stress/low stress/medium stress/high stress), and/or anumerical value (e.g., a value on a scale of 0 to 10). In someembodiments, a “stress level” may refer to a “fight or flight” reactionlevel.

Evaluation of stress typically involves determining an amount of stressa person may be feeling according to some standard scale. There arevarious approaches known in the literature that may be used for thistask. One approach involves identifying various situations the personmay be in, which are associated with certain predefined extents ofstress (which are empirically derived based on observations). Example ofpopular approaches include the Holmes and Rahe stress scale, thePerceived Stress Scale, and the Standard Stress Scale (SSS). A commontrait of many the various stress scales is that they require a manualevaluation of situations a user undergoes, and do not measure the actualphysiological effects of stress.

In some embodiments, the computer may receive an indication of a type ofstressor, and utilize the indication to detect the stress level.Optionally, the indication is indicative of a period and/or durationduring which the user was affected by the stressor. In one example, theindication is utilized to select a certain threshold value, which isappropriate for the type of stressor, and to which TH_(ROI1) may becompared in order to determine whether the user is experiencing acertain stress level. Optionally, the certain threshold is determinedbased on thermal measurements of the user when the user reacted to astressor of the indicated type. In another example, the indication isutilized to select a certain reference time series, which corresponds tothe type of stressor, and to which TH_(ROI1) may be compared in order todetermine whether the user is experiencing a certain stress level.Optionally, the certain time series is based on thermal measurements ofthe user taken when the user reacted to a stressor of the indicatedtype. In yet another example, the computer generates one or more featurevalues based on the indication, and the one or more feature values areutilized to detect the stress level using a model (in addition tofeature values generated based on TH_(ROI1)). In still another example,the computer may select a window of time based on the indication, whichcorresponds to the expected duration of stress induced by the type ofstressor indicated in the indication. In this example, in order todetect the stress level of the user, the computer evaluates thermalmeasurements from among TH_(ROI1) that were taken at a time that fallsin the window.

Additional CAMs may be utilized to detect the stress level. The thermalmeasurements of the additional CAMs, typically denoted TH_(ROI2) below,may be utilized to generate one or more of the feature values that areused along with the machine learning-based model to detect the stresslevel.

In one embodiment, the system includes a second inward-facinghead-mounted thermal camera (CAM2) that takes thermal measurements of anadditional ROI on the face (TH_(ROI2)), such as the forehead, the nose,and/or a region below the nostrils. The region below the nostrils referto one or more regions on the upper lip, the mouth, and/or air volumethrough which the exhale streams from the nose and/or mouth flow, andit's thermal measurements are indicative of the user's breathing.

Given TH_(ROI2), the computer may generate feature values based onTH_(ROI1) and TH_(ROI2) (and possibly other sources of data) andutilizes a model to detect the stress level based on the feature values.Optionally, the model was trained based on previous TH_(ROI1) andTH_(ROI2) taken while the user had at least two different stress levelsaccording to a predetermined stress scale. For example, a first set ofprevious TH_(ROI1) and TH_(ROI2) taken while the user was under elevatedstress, and a second set of previous TH_(ROI1) and TH_(ROI2) taken whilethe user was not under elevated stress.

In another embodiment, the system further includes second and third CAMsthat take thermal measurements of regions on the right and left cheeks,respectively. Optionally, the computer detects the stress level alsobased on the thermal measurements of the cheeks.

FIG. 23 illustrates one embodiment of an HMS able to measure stresslevel. The system includes a frame 51, CAMs (52, 53, 54), and a computer56. CAMs are physically coupled to the frame and take thermalmeasurements of ROIs on the periorbital areas. Because CAMs are locatedclose to their respective ROIs, they can be small, lightweight, and maybe placed in many potential locations having line of sight to theirrespective ROIs. The computer 56, which may by located on the HMS, wornby the user, and/or remote such as in the cloud, detects the stresslevel based on changes to temperature of the periorbital areas receivedfrom the CAMs.

Due to the asymmetry of blood vessels in human faces and differentshapes of human faces, having CAMs pointed at the right and leftperiorbital areas may enable a more accurate detection of physiologicalphenomena such as stress, and/or may enable detection of stress that isharder to detect based on measuring only a single periorbital area.

While FIG. 23 and FIG. 24 illustrate examples of asymmetric locations ofCAMs that measure the right periorbital area relative to the locationsof CAMs that measure the left periorbital area, FIG. 25 illustrates anexample of symmetric locations of the CAMs that measure the rightperiorbital area relative to the locations of the CAMs that measure theleft periorbital area. In some embodiments, using thermal measurementsfrom both symmetric and asymmetric located CAMs may improve the system'sadaptability to different faces having different proportions.

FIG. 27a and FIG. 27b illustrate one scenario of detecting a user'sstress level. FIG. 27a illustrates a child watching a movie whilewearing an eyeglasses frame 570 with at least five CAMs. FIG. 27billustrates the at least five CAMs 571, 572, 573, 574, and 575, whichmeasure the right and left periorbital areas, the nose, and the rightand left cheeks, respectively (the different ROIs are designated bydifferent patterns). The figure further illustrates how the systemproduces a graph of the stress level detected at different times whiledifferent movie scenes were viewed.

In one embodiment, the system may include a head-mounted display (HMD)that presents digital content to the user and does not prevent CAM frommeasuring the ROI. In another embodiment, the system may include an eyetracker to track the user's gaze, and an optical see through HMD thatoperates in cooperation with the following components; a visible-lightcamera that captures images of objects the user is looking at, and acomputer that matches the objects the user is looking at with thedetected stress levels. Optionally, the eye tracker is coupled to aframe worn by the user. In yet another embodiment, the system mayinclude a HMD that presents video comprising objects, and an eyetracker. The computer utilizes data generated by the eye tracker tomatch the objects the user is looking at with the detected stresslevels. It is to be noted that there may be a delay between beingaffected by a stressor and a manifestation of stress as a reaction, andthis delay may be taken into account when determining what objectscaused the user stress.

In one embodiment, the system further includes a user interface (UI),such as user interface 483 illustrated in FIG. 29, which notifies theuser when the stress level reaches a predetermined threshold.Optionally, the UI notifies the user by an audio indication, a visualindication, and/or a haptic notification. Optionally, the greater thechange to the temperature of the periorbital areas, the higher thedetected stress level, and the indication is proportional to the stresslevel.

FIG. 26 illustrates a scenario in which a system (which measures theforehead, right and left periorbital areas, nose, and below thenostrils) suggests to the user to take a break in order to reduce thestress level of the user. The system may suggest the user to partake inat least one of the following activities when the stress level reaches afirst threshold: practice pranayama, physical exercise, listen tobrainwave entrainment, and listen to positive loving statements.Optionally, the computer suggests to the user to stop the activity whenthe stress level gets below a second threshold. Optionally, the systemshows the user video comprising objects, and the detected stress levelassociated with the objects.

FIG. 28 illustrates one embodiment of a system configured to generate apersonalized model for detecting stress based on thermal measurements ofthe face. The system includes a frame, first and second CAMs, and acomputer 470. The first and second CAMs take thermal measurements 471 ofregions on the periorbital areas of the right and left eyes (TH_(ROI1)and TH_(ROI2), respectively) of the user 472.

The computer 470 generates samples based on data comprising: (i)TH_(ROI1) and TH_(ROI2) 471, and (ii) indications 473 corresponding todifferent times, which are indicative of stress levels of the user atthe different times. Optionally, each sample comprises: (i) featurevalues generated values based on TH_(ROI1) and TH_(ROI2) taken during acertain period, and (ii) a label indicative of a stress level of theuser during the certain period. Optionally, at least one of the featurevalues in a sample may be generated based on other sources ofinformation such as physiological measurements of the user 472 and/ormeasurements of the environment in which the user 472 was in when whileTH_(ROI1) and TH_(ROI2) 471 were taken. Optionally, the stress levelsindicated in the indications 473 correspond to levels of a known stresslevel scale. The computer 470 trains a model 477 based on the samples.Optionally, the computer 470 also provides the model 477 to be used by asystem that detects stress based on TH_(ROI1) and TH_(ROI2).

The indications 473 may be generated in different ways, in differentembodiments. One or more of the indications 473 may be (i) generated byan entity that observes the user 472, such as a human observer or asoftware program (e.g., a software agent operating on behalf of the user472), (ii) provided by the user 472, such as via a smartphone app bypressing a certain button on a screen of a smartphone, and/or by speechthat is interpreted by a software agent and/or a program with speechanalysis capabilities, (iii) determined based on analysis of behavior ofthe user 472, such as by analyzing measurements of a camera and/or amicrophone that indicate that the user 472 is experiencing a certainstress level, and (iv) determined based on physiological signals of theuser 472 that are not thermal measurements of one or more ROIs on theface, such as measurements of the user's heart rate and/or brainwaveactivity.

Optional stress analysis module 497 receives descriptions of eventscorresponding to when at least some of TH_(ROI1) and TH_(ROI2) 471 weretaken, and generates one or more of the indications 473 based onanalyzing the descriptions. The stress analysis module 497 isimplemented by the computer 470 or another computer. Optionally, all ofthe indications 473 are generated by the stress analysis module 497.Optionally, the stress analysis module 497 may be a module of a softwareagent operating on behalf of the user 472. The descriptions received bythe stress analysis module 497 may include various forms of information.In one example, the descriptions include content of a communication ofthe user 472, and the stress analysis module 497 utilizes semanticanalysis in order to determine whether the communication is indicative astressful event for the user 472 (e.g., the communication is indicativeof something going wrong at work). Optionally, the stress analysismodule 497 utilizes a machine learning-based model to calculate based onfeatures derived from the communication, a predicted stress level forthe user 472. In another example, the stress analysis module 497receives images of an event, such as images taken by an outward-facinghead-mounted visible-light camera, utilizes image analysis to determinewhether the event corresponds to a stressful event, and utilizes amachine learning-based model to calculate the predicted stress based onfeatures derived from the images.

The model is trained on samples comprising feature values based onTH_(ROI1) and TH_(ROI2), and additional feature values described in thefollowing examples:

In a first example, the additional feature values include additionalthermal measurements, taken with another CAM, of an ROI that includesthe nasal and/or mouth regions.

In a second example, the additional feature values are indicative of oneor more of the following signals of the user 472; a heart rate, heartrate variability, brainwave activity, galvanic skin response, muscleactivity, and an extent of movement.

In a third example, the additional feature values are measurements(m_(conf) 474) of the user 472 and/or of the environment in which theuser 472 was in while TH_(ROI1) and TH_(ROI2) 471 were taken.Optionally, mar 474 are taken by a sensor 461, which may be physicallycoupled to the frame. In another example, the sensor 461 is coupled to adevice carried by the user, such as a smartphone, a smartwatch, and/orsmart clothing (e.g., clothing embedded with sensors that can measurethe user and/or the environment). In yet another example, the sensor 461may be an external sensor that is not carried by the user. Optionally,the computer 470 is generates, based on m_(conf) 474, one or morefeature values of at least some of the samples. m_(conf) 474 areindicative of an extent to which one or more confounding factorsoccurred while TH_(ROI1) and TH_(ROI2) 471 were taken.

In one embodiment, the sensor 461 is a visible-light camera physicallycoupled to the frame, which takes images of a region on the face of theuser 472, which includes at least 25% of the ROI₁ and/or ROI₂.Optionally, the confounding factor in this embodiment involvesinflammation of the skin, skin blemishes, food residues on the face,talking, eating, drinking, and/or touching the face. In anotherembodiment, the sensor 461 includes a movement sensor that measures amovement of the user 472. Optionally, the confounding factor in thisembodiment involves the user 472 walking, running, exercising, bendingover, and/or getting up from a sitting or lying position. In yet anotherembodiment, the sensor 461 measures at least one of the followingenvironmental parameters: a temperature of the environment, a humiditylevel of the environment, a noise level of the environment, air qualityin the environment, a wind speed in the environment, an extent ofprecipitation in the environment, and an infrared radiation level in theenvironment.

Some aspects of this disclosure involve monitoring a user over time withCAM that takes thermal measurements of a region on a periorbital area(TH_(ROI1)) of the user. One application for which TH_(ROI1) may beuseful is to detect the stress level of the user. Analysis of thesedetections combined with information regarding factors that affected theuser at different times, which may be considered potential stressors,can reveal which of the factors may be stressor that increase the stresslevel of the user.

Some examples of factors that may be considered potential stressors forcertain users include being in certain locations, interacting withcertain entities, partaking in certain activities, or being exposed tocertain content. Having knowledge of which potential stressor are likelyto actually be stressors for a certain user can help that user avoid thestressors and/or take early measures to alleviate the effects of thestress they cause.

FIG. 30 illustrates one embodiment of a system configured to select astressor. The system includes at least a computer 486 and CAM. Thesystem may optionally include a frame 469, a camera 383, and/or a UI495. In one example, CAM takes thermal measurements of the periorbitalarea of the right eye, and an additional CAM (CAM2) takes thermalmeasurements of a region on the periorbital area of the left eye(TH_(ROI2)) of the user.

In one embodiment, computer 486 calculates, based on the thermalmeasurements 487 (e.g., TH_(ROI1) and TH_(ROI2)), values that areindicative of stress levels of the user at different times (i.e., detectthe stress levels of the user at the different times). Optionally,TH_(ROI1) and TH_(ROI2) include thermal measurements taken while theuser had at least two different stress levels according to apredetermined stress scale. Optionally, the thermal measurements 487comprise thermal measurements taken during different days.

In some embodiments, the system that selects a stressor may includeadditional CAMs that take thermal measurements of one or more regions onthe user's forehead, nose, and/or below the nostrils. Optionally,thermal measurements taken by the additional CAMs are utilized by thecomputer 486 when calculating the user's stress level.

Furthermore, the computer 486 may receive indications 490 of factorsthat affected the user at various times, which may be consideredpotential stressors. The computer 486 also selects a stressor 491, fromamong the potential stressors, based on the indications 490 and thevalues that are indicative of stress levels of the user at differenttimes. Optionally, each of the indications 490 is indicative of a timeduring which the user was exposed to a potential stressor. Additionallyor alternatively, each of the indications 490 may be indicative of atime during which the user was affected by a potential stressor. In someembodiments, at any given time, the user may be exposed to more than oneof the potential stressors. Thus, in some embodiments, at least some ofthe thermal measurements 487, and optionally all of the thermalmeasurements 487, were taken while the user was exposed to two or morepotential stressors.

In one embodiment, the indications 490 include a list of periods of timeduring which various potential stressors affected the user. Optionally,the indications 490 are provided via a data structure and/or a queryablesystem that provides information for different points in time aboutwhich of the potential stressors affected the user at the points intime. There are various types of potential stressors that may beindicated by the indications 490.

In one embodiment, one or more of the potential stressors may relate tovarious locations the user was at (e.g., work, school, doctor's office,in-laws house, etc.) and/or to various activities the user partakes in(e.g., driving, public speaking, operating machinery, caring forchildren, choosing clothes to wear, etc.)

In another embodiment, one or more of the potential stressors may relateto entities with which the user interacts. For example, an entity may bea certain person, a person with a certain role (e.g., a teacher, apolice officer, a doctor, etc.), a certain software agent, and/or anavatar (representing a person or a software agent).

In yet another embodiment, one or more of the potential stressors mayrelate to situations in which the user is in, which can increase stress.For example, a situation may be being unemployed, having financialdifficulties, being separated after being in a relationship with anotherperson, being alone, or awaiting an important event (e.g., an exam, ajob interview, or results of an important medical test). In anotherexample, a situation may relate to a physical condition of the user,such as being sick or suffering from a certain chronic disease.Optionally, when the situations described above are applicable toanother person who the user cares about (e.g., a spouse, child, parent,or close friend), then those situations, which relate to the otherperson, may be considered potential stressors that can lead to stress inthe user.

In still another embodiment, one or more of the potential stressors mayrelate to the user's behavior. For example, behaving in a way that isargumentative, manipulative, deceptive, and/or untruthful may increasethe stress level.

There are various ways in which the computer 486 may select, based onthe indications 490 and the thermal measurements 487, the stressor 491from among the potential stressors being considered.

In some embodiments, the computer 486 performs a direct analysis of theeffect of each of the potential stressors in order to identify whichones have a large effect on the user. Optionally, the effect of eachpotential stressor is indicative of the extent to which it increases thestress level of the user. Optionally, the effect of each potentialstressor is calculated by determining, based on the indications 490,times at which the user was affected by the potential stressor, andobserving the stress level of the user at one or more times that are upto a certain period Δ later (where Δ depends on the user and the type ofstressor). In one example, Δ is ten seconds, thirty seconds, or oneminute. In another example, Δ is one minute, ten minutes, or one hour.

In one embodiment, a stress level (or change to the stress level)following being affected by a potential stressor is the maximum stresslevel that is detected from the time t the user was affected by thepotential stressor until the time t+Δ. In another example, the stresslevel (or change to the stress level) following being affected by thepotential stressor is the extent of the stress level and/or change tothe stress level that is detected at a time t+Δ (when the user wasaffected by the potential stressor at time t). Optionally, the extentmay be normalized based on a quantitative value representing how muchthe user was affected by the potential stressor. Optionally, the stresslevel may be normalized with respect to a stress level detected prior tobeing affected by the potential stressor.

Following a calculation of the effects of the potential stressors, inone embodiment, the computer 486 selects the stressor 491 from among thepotential stressors. Optionally, the stressor 491 is a potentialstressor that has a maximal effect (i.e., there is no other potentialstressor that has a higher effect). Optionally, the stressor 491 is apotential stressor that has an effect that reaches a threshold, whilethe effects of most of the potential stressors do not reach thethreshold.

In one embodiment, in order to increase confidence in the selection ofthe stressor, the stressor 491 is selected based on at least a certainnumber of times in which the user was affected by the stressor 491. Forexample, the certain number may be at least 3 or 10 different times.Thus, in this embodiment, potential stressors that did not affect theuser at least the certain number of times are not selected.

In some embodiments, the computer 486 generates a machine learning-basedmodel based on the indications 490 and the values indicative of thestress levels of the user, and selects the stressor 491 based on ananalysis of the model. Optionally, the computer 486 generates samplesused to train the model. The samples used to train the model maycorrespond to different times, with each sample corresponding to a timet+Δ including feature values and a label indicative of the stress levelof the user at the time t+Δ. Each sample may be considered to representa snapshot of potential stressors that affected the user during acertain period, and a label that is indicative of the stress level ofthe user following being affected by those potential stressors. Givenmultiple such samples, a machine learning training algorithm can beutilized to train a model for a predictor module that can predict thestress level at a certain time based on feature values that describepotential stressors that affected the user during a certain period oftime leading up to the certain time. For example, if the model is aregression model, the predictor module may perform a dot productmultiplication between a vector of regression coefficients (from themodel) and a vector of the feature values in order to calculate a valuecorresponding to the predicted stress level of the user at the certaintime.

When such a predictor module is capable of predicting stress level ofthe user based on the feature values described above, this may mean thatthe model captures, at least to some extent, the effects of at leastsome of the potential stressors on the stress level of the user.

Training the model based on the samples described above may involveutilizing one or more of various training algorithms. Some examples ofmodels that may be generated in order to be utilized by the predictormodule described above include the following models; a regression model(e.g., a regression model), a naïve Bayes model, a Bayes network, asupport vector machine for regression, a decision tree, and a neuralnetwork model, to name a few possibilities. There are various trainingalgorithms known in the art for generating these models and other modelswith similar properties.

The predictor module may be provided multiple inputs representing thepotential stressors that affected the user at different points of time,and have a capability to store state information of previous inputscorresponding to earlier times when it comes to predict the stress levelof the user at a certain time. For example, the predictor module may bebased on a recurrent neural network.

Once the model is trained, in some embodiments, it is analyzed by thecomputer 486 in order to determine the effects of one or more of thestressors on the stress level of the user. Depending on the type ofmodel that was trained, this analysis may be performed in differentways.

In one embodiment, the computer 486 performs the analysis of the modelby evaluating parameters of the model that correspond to the potentialstressors. Optionally, the computer 486 selects as the stressor 491 acertain potential stressor that has a corresponding parameter that isindicative of an effect that reaches a threshold while effects indicatedin parameters corresponding to most of the stressors do not reach thethreshold. In one example, the model may be a linear regression model inwhich each potential stressor corresponds to a regression variable. Inthis example, a magnitude of a value of a regression coefficient may beindicative of the extent of the effect of its corresponding potentialstressor. In another example, the model may be a naïve Bayes model inwhich various classes correspond to stress levels (e.g., a binaryclassification model that is used to classify a vector of feature valuesto classes corresponding to “stressed” vs. “not stressed”). In thisexample, each feature value may correspond to a potential stressor, andthe class conditional probabilities in the model are indicative of theeffect of each of the potential stressors on the user.

In another embodiment, the computer 486 performs an analysis of themodel, which may be characterized as “black box” analysis. In thisapproach, the predictor module is provided with various inputs thatcorrespond to different potential stressors that affect the user, andcalculates, based on the inputs and the model, various predicted stresslevels of the user. The various inputs can be used to independentlyand/or individually increase the extent to which each of the potentialstressors affects the user. This type of the model probing can helpidentify certain potential stressors that display an increase in thepredicted stress level, which corresponds to an increase in the extentto which the potential stressors affect the user (according to themodel). Optionally, the stressor 491 is a potential stressor for which apositive correlation is observed between increasing the extent to whichthe potential stressor affects that user, and the predicted stress levelof the user. Optionally, the stressor 491 is selected from among thepotential stressors, responsive to identifying that: (i) based on afirst subset of the various predicted stress levels of the user, aneffect of the stressor 491 reaches a threshold, and (ii) based on asecond subset of the various predicted stress levels of the user,effects of most of the potential stressors do not reach the threshold.

The indications 490 may be received from various sources. In oneembodiment, the user may provide at least some of the indications 490(e.g., by inputting data via an app and/or providing vocal annotationsthat are interpreted by a speech analysis software). In otherembodiments, at least some of the indications 490 are provided byanalysis of one or more sources of data. Optionally, the computer 486generates one or more of the indications 490 based on an analysis ofdata obtained from the one or more sources. The following four examples,discussed herein in relation to allergy, are also relevant as examplesof sources of data that may be utilized to identify potential stressorsthat affected the user at different times: (i) a camera 383 capturesimages of the surroundings of the user, (ii) sensors such asmicrophones, accelerometers, thermometers, pressure sensors, and/orbarometers may be used to identify potential stressors by identifyingwhat the user is doing and/or under what conditions, (iii) measurementsof the environment that user is in, and (iv) IoT devices, communicationsof the user, calendar, and/or billing information may provideinformation that may be used in some embodiments to identify potentialstressors.

With little modifications, the system illustrated in FIG. 30 may beutilized to detect a calming factor that reduces the user's stress,rather than one that increases it. In particular, instead of selecting astressor that has a large effect (or maximal effect) on the user, afactor that has a large negative effect on the stress level may beselected. Optionally, in the event that a high stress level of the useris detected, the calming factor may be suggested to the user (to reducethe user's stress level).

When mounting a camera having a large field of view in sharp angle andclose to the face, the captured image is usually not sharp all overbecause the object is not parallel to the sensor plane. There is a needto improve the quality of the images obtained from a camera mounted inclose proximity and sharp angle to the face.

Normally, the lens plane and the sensor plane of a camera are parallel,and the plane of focus (PoF) is parallel to the lens and sensor planes.If a planar object is also parallel to the sensor plane, it can coincidewith the PoF, and the entire object can be captured sharply. If the lensplane is tilted (not parallel) relative to the sensor plane, it will bein focus along a line where it intersects the PoF. The Scheimpflugprinciple is a known geometric rule that describes the orientation ofthe plane of focus of a camera when the lens plane is tilted relative tothe sensor plane.

FIG. 20a is a schematic illustration of an inward-facing head-mountedcamera 550 embedded in an eyeglasses frame 551, which utilizes theScheimpflug principle to improve the sharpness of the image taken by thecamera 550. The camera 550 includes a sensor 558 and a lens 555. Thetilt of the lens 555 relative to sensor 558, which may also beconsidered as the angle between the lens plane 555 and the sensor plane559, is determined according to the expected position of the camera 550relative to the ROI 552 when the user wears the eyeglasses. For arefractive optical lens, the “lens plane” 556 refers to a plane that isperpendicular to the optical axis of the lens 555. Herein, the singularalso includes the plural, and the term “lens” refers to one or morelenses. When “lens” refers to multiple lenses (which is usually the casein most modern cameras having a lens module with multiple lenses), thenthe “lens plane” refers to a plane that is perpendicular to the opticalaxis of the lens module.

The Scheimpflug principle may be used for both thermal cameras (based onlenses and sensors for wavelengths longer than 2500 nm) andvisible-light and/or near-IR cameras (based on lenses and sensors forwavelengths between 400-900 nm). FIG. 20b is a schematic illustration ofa camera that is able to change the relative tilt between its lens andsensor planes according to the Scheimpflug principle. Housing 311 mountsa sensor 312 and lens 313. The lens 313 is tilted relative to the sensor312. The tilt may be fixed according to the expected position of thecamera relative to the ROI when the user wears the HMS, or may beadjusted using motor 314. The motor 314 may move the lens 313 and/or thesensor 312.

In one embodiment, an HMS device includes a frame configured to be wornon a user's head, and an inward-facing camera physically coupled to theframe. The inward-facing camera may assume one of two configurations:(i) the inward-facing camera is oriented such that the optical axis ofthe camera is above the Frankfort horizontal plane and pointed upward tocapture an image of a region of interest (ROI) above the user's eyes, or(ii) the inward-facing camera is oriented such that the optical axis isbelow the Frankfort horizontal plane and pointed downward to capture animage of an ROI below the user's eyes. The inward-facing camera includesa sensor and a lens. The sensor plane is tilted by more than 2° relativeto the lens plane according to the Scheimpflug principle in order tocapture a sharper image.

In another embodiment, an HMS includes an inward-facing head-mountedcamera that captures an image of an ROI on a user's face, when worn onthe user's head. The ROI is on the user's forehead, nose, upper lip,cheek, and/or lips. The camera includes a sensor and a lens. And thesensor plane is tilted by more than 2° relative to the lens planeaccording to the Scheimpflug principle in order to capture a sharperimage.

Because the face is not planar and the inward-facing head-mounted camerais located close to the face, an image captured by a camera having awide field of view (FOV) and a low f-number may not be perfectly sharp,even after applying the Scheimpflug principle. Therefore, in someembodiments, the tilt between the lens plane and the sensor plane isselected such as to adjust the sharpness of the various areas covered inthe ROI according to their importance for detecting the user'sphysiological response (which may be the user's emotional response insome cases). In one embodiment, the ROI covers first and second areas,where the first area includes finer details and/or is more important fordetecting the physiological response than the second area. Therefore,the tilt between the lens and sensor planes is adjusted such that theimage of the first area is shaper than the image of the second area.

In another embodiment, the ROI covers both a first area on the upper lipand a second area on a cheek, and the tilt is adjusted such that theimage of the first area is shaper than the image of the second area,possibly because the upper lip usually provides more information and hasmore details relative to the cheek.

In still another embodiment, the ROI covers both a first area on theupper lip and a second area on the nose, and the tilt is adjusted suchthat the image of the first area is shaper than the image of the secondarea, possibly because the upper lip usually provides more informationrelative to the nose.

In still another embodiment, the ROI covers a first area on the cheekstraight above the upper lip, a second area on the cheek from the edgeof the upper lip towards the ear, and a third area on the nose. And thetilt between the lens plane and the sensor plane is adjusted such thatthe image of the first area is shaper than both the images of the secondand third areas.

In still another embodiment, the ROI covers both a first area on thelips and a second area on the chin, and the tilt is adjusted such thatthe image of the first area is shaper than the image of the second area,possibly because the lips usually provides more information than thechin.

In still another embodiment, the camera is a visible-light camera, andthe ROI covers both a first area on the lower forehead (including aneyebrow) and a second area on the upper forehead, and the tilt isadjusted such that the image of the first area is shaper than the imageof the second area, possibly because the eyebrow provides moreinformation about the user's emotional response than the upper forehead.

In still another embodiment, the camera is a thermal camera, and the ROIcovers an area on the forehead, and the tilt is adjusted such that theimage of a portion of the middle and upper part of the forehead (belowthe hair line) is shaper than the image of a portion of the lower partof the forehead, possibly because the middle and upper parts of theforehead are more indicative of prefrontal cortex activity than thelower part of the forehead, and movements of the eyebrows disturb thethermal measurements of the lower part of the forehead.

In one embodiment, the tilt between the lens plane and sensor plane isfixed. The fixed tilt is selected according to an expected orientationbetween the camera and the ROI when a user wears the frame. Having afixed tilt between the lens and sensor planes may eliminate the need foran adjustable electromechanical tilting mechanism. As a result, a fixedtilt may reduce the weight and cost of the camera, while still providinga sharper image than an image that would be obtained from a similarcamera in which the lens and sensor planes are parallel. The magnitudeof the fixed tilt may be selected according to facial dimensions of anaverage user expected to wear the system, or according to a model of thespecific user expected to wear the system in order to obtain thesharpest image.

In another embodiment, the system includes an adjustableelectromechanical tilting mechanism configured to change the tiltbetween the lens and sensor planes according to the Scheimpflugprinciple based on the orientation between the camera and the ROI whenthe frame is worn by the user. The tilt may be achieved using at leastone motor, such as a brushless DC motor, a stepper motor (without afeedback sensor), a brushed DC electric motor, a piezoelectric motor,and/or a micro-motion motor.

The adjustable electromechanical tilting mechanism configured to changethe tilt between the lens and sensor planes may include one or more ofthe following mechanisms: (i) a mirror that changes its angle: (ii) adevice that changes the angle of the lens relative to the sensor; and/or(iii) a device that changes the angle of the sensor relative to thelens. In one embodiment, the camera, including the adjustableelectromechanical tilting mechanism, weighs less than 10 g, and theadjustable electromechanical tilting mechanism is able to change thetilt in a limited range below 30° between the two utmost orientationsbetween the lens and sensor planes. Optionally, the adjustableelectromechanical tilting mechanism is able to change the tilt in alimited range below 20° between the two utmost orientations between thelens and sensor planes. In another embodiment, the adjustableelectromechanical tilting mechanism is able to change the tilt in alimited range below 10°. In some embodiments, being able to change thetilt in a limited range reduces at least one of the weight, cost, andsize of the camera, which is advantageous for a wearable device. In oneexample, the camera is manufactured with a fixed predetermined tiltbetween the lens and sensor planes, which is in addition to the tiltprovided by the adjustable electromechanical tilting mechanism. Thefixed predetermined orientation may be determined according to theexpected orientation between the camera and the ROI for an average user,such that the adjustable electromechanical tilting mechanism is used tofine-tune the tilt between the lens and sensor planes for the specificuser who wears the frame and has facial dimensions that are differentfrom the average user.

Various types of cameras may be utilized in different embodimentsdescribed herein. In one embodiment, the camera is a thermal camera thattakes thermal measurements of the ROI with a focal plane array thermalsensor having an angle above 2° between the lens and sensor planes.Optionally, the thermal camera weighs below 10 g, is located less than10 cm from the user's face, and the tilt of the lens plane relative tothe sensor plane is fixed. The fixed tilt is selected according to anexpected orientation between the camera and the ROI when the user wearsthe frame. Optionally, the system includes a computer to detect aphysiological response based on the thermal measurements. Optionally,the computer processes time series measurements of each sensing elementindividually to detect the physiological response.

In another embodiment, the camera is a visible-light camera that takesvisible-light images of the ROI, and a computer generates an avatar forthe user based on the visible-light images. Some of the variousapproaches that may be utilized to generate the avatar based on thevisible-light images are described in co-pending US patent publication2016/0360970. Additionally or alternatively, the computer may detect anemotional response of the user based on (i) facial expressions in thevisible-light images utilizing image processing, and/or (ii) facial skincolor changes (FSCC), which result from concentration changes ofhemoglobin and/or oxygenation.

It is to be noted that there are various approaches known in the art foridentifying facial expressions from images. While many of theseapproaches were originally designed for full-face frontal images, thoseskilled in the art will recognize that algorithms designed for full-facefrontal images may be easily adapted to be used with images obtainedusing the inward-facing head-mounted visible-light cameras disclosedherein. For example, the various machine learning techniques describedin prior art references may be applied to feature values extracted fromimages that include portions of the face from orientations that are notdirectly in front of the user. Furthermore, due to the closeness of thevisible-light cameras to the face, facial features are typically largerin images obtained by the systems described herein. Moreover, challengessuch as image registration and face tracking are vastly simplified andpossibly non-existent when using inward-facing head-mounted cameras. Thereference Zeng, Zhihong, et al. “A survey of affect recognition methods:Audio, visual, and spontaneous expressions.” IEEE transactions onpattern analysis and machine intelligence 31.1 (2009): 39-58, describessome of the algorithmic approaches that may be used for this task. Thefollowing references discuss detection of emotional responses based onFSCC: (i) Ramirez, Geovany A., et al. “Color analysis of facial skin:Detection of emotional state” in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition Workshops, 2014; and (ii) Wang.Su-Jing, et al. “Micro-expression recognition using color spaces”, inIEEE Transactions on Image Processing 24.12 (2015): 6034-6047.

In still another embodiment, the camera is a light field camera thatimplements a predetermined blurring at a certain Scheimpflug angle, anddecodes the predetermined blurring as function of the certainScheimpflug angle. The light field camera may include an autofocusing ofthe image obtained using the tilting mechanism based on the principlethat scene points that are not in focus are blurred while scene pointsin focus are sharp. The autofocusing may study a small region around agiven pixel; the region is expected to get sharper as the Scheimpflugadjustment gets better, and vice versa. Additionally or alternatively,the autofocusing may use the variance of the neighborhood around eachpixel as a measure of sharpness, where a proper Scheimpflug adjustmentshould increase the variance.

Various embodiments described herein involve an HMS that may beconnected, using wires and/or wirelessly, with a device carried by theuser and/or a non-wearable device. The HMS may include a battery, acomputer, sensors, and a transceiver.

FIG. 31a and FIG. 31b are schematic illustrations of possibleembodiments for computers (400, 410) that are able to realize one ormore of the embodiments discussed herein that include a “computer”. Thecomputer (400, 410) may be implemented in various ways, such as, but notlimited to, a server, a client, a personal computer, a network device, ahandheld device (e.g., a smartphone), an HMS (such as smart glasses, anaugmented reality system, and/or a virtual reality system), a computingdevice embedded in a wearable device (e.g., a smartwatch or a computerembedded in clothing), a computing device implanted in the human body,and/or any other computer form capable of executing a set of computerinstructions. Herein, an augmented reality system refers also to a mixedreality system. Further, references to a computer or processor includeany collection of one or more computers and/or processors (which may beat different locations) that individually or jointly execute one or moresets of computer instructions. For example, a first computer may beembedded in the HMS that communicates with a second computer embedded inthe user's smartphone that communicates over the Internet with a cloudcomputer.

The computer 400 includes one or more of the following components:processor 401, memory 402, computer readable medium 403, user interface404, communication interface 405, and bus 406. The computer 410 includesone or more of the following components: processor 411, memory 412, andcommunication interface 413.

Thermal measurements that are forwarded to a processor/computer mayinclude “raw” values that are essentially the same as the valuesmeasured by thermal cameras, and/or processed values that are the resultof applying some form of preprocessing and/or analysis to the rawvalues. Examples of methods that may be used to process the raw valuesinclude analog signal processing, digital signal processing, and variousforms of normalization, noise cancellation, and/or feature extraction.

At least some of the methods described herein are “computer-implementedmethods” that are implemented on a computer, such as the computer (400,410), by executing instructions on the processor (401, 411). Optionally,the instructions may be stored on a computer-readable medium, which mayoptionally be a non-transitory computer-readable medium. In response toexecution by a system including a processor and memory, the instructionscause the system to perform the method steps.

Herein, a direction of the optical axis of a VCAM or a CAM that hasfocusing optics is determined by the focusing optics, while thedirection of the optical axis of a CAM without focusing optics (such asa single pixel thermopile) is determined by the angle of maximumresponsivity of its sensor. When optics are utilized to takemeasurements with a CAM, then the term CAM includes the optics (e.g.,one or more lenses). In some embodiments, the optics of a CAM mayinclude one or more lenses made of a material suitable for the requiredwavelength, such as one or more of the following materials: CalciumFluoride, Gallium Arsenide, Germanium, Potassium Bromide, Sapphire,Silicon, Sodium Chloride, and Zinc Sulfide. In other embodiments, theCAM optics may include one or more diffractive optical elements, and/oror a combination of one or more diffractive optical elements and one ormore refractive optical elements.

When CAM includes an optical limiter/field limiter/FOV limiter (such asa thermopile sensor inside a standard TO-39 package with a window, or athermopile sensor with a polished metal field limiter), then the termCAM may also refer to the optical limiter. Depending on the context, theterm CAM may also refer to a readout circuit adjacent to CAM, and/or tothe housing that holds CAM.

Herein, references to thermal measurements in the context of calculatingvalues based on thermal measurements, generating feature values based onthermal measurements, or comparison of thermal measurements, relate tothe values of the thermal measurements (which are values of temperatureor values of temperature changes). Thus, a sentence in the form of“calculating based on TH_(ROI)” may be interpreted as “calculating basedon the values of TH_(ROI)”, and a sentence in the form of “comparingTH_(ROI1) and TH_(ROI2)” may be interpreted as “comparing values ofTH_(ROI1) and values of TH_(ROI2)”.

Depending on the embodiment, thermal measurements of an ROI (usuallydenoted TH_(ROI) or using a similar notation) may have various forms,such as time series, measurements taken according to a varying samplingfrequency, and/or measurements taken at irregular intervals. In someembodiments, thermal measurements may include various statistics of thetemperature measurements (T) and/or the changes to temperaturemeasurements (ΔT), such as minimum, maximum, and/or average values.Thermal measurements may be raw and/or processed values. When a thermalcamera has multiple sensing elements (pixels), the thermal measurementsmay include values corresponding to each of the pixels, and/or includevalues representing processing of the values of the pixels. The thermalmeasurements may be normalized, such as normalized with respect to abaseline (which is based on earlier thermal measurements), time of day,day in the month, type of activity being conducted by the user, and/orvarious environmental parameters (e.g., the environment's temperature,humidity, radiation level, etc.).

As used herein, references to “one embodiment” (and its variations) meanthat the feature being referred to may be included in at least oneembodiment of the invention. Moreover, separate references to “oneembodiment”, “some embodiments”, “another embodiment”, “still anotherembodiment”, etc., may refer to the same embodiment, may illustratedifferent aspects of an embodiment, and/or may refer to differentembodiments.

Some embodiments may be described using the verb “indicating”, theadjective “indicative”, and/or using variations thereof. Herein,sentences in the form of “X is indicative of Y” mean that X includesinformation correlated with Y, up to the case where X equals Y. Forexample, sentences in the form of “thermal measurements indicative of aphysiological response” mean that the thermal measurements includeinformation from which it is possible to infer the physiologicalresponse. Stating that “X indicates Y” or “X indicating Y” may beinterpreted as “X being indicative of Y”. Additionally, sentences in theform of “provide/receive an indication indicating whether X happened”may refer herein to any indication method, including but not limited to:sending/receiving a signal when X happened and not sending/receiving asignal when X did not happen, not sending/receiving a signal when Xhappened and sending/receiving a signal when X did not happen, and/orsending/receiving a first signal when X happened and sending/receiving asecond signal X did not happen.

Herein, “most” of something is defined as above 51% of the something(including 100% of the something). Both a “portion” of something and a“region” of something refer herein to a value between a fraction of thesomething and 100% of the something. For example, sentences in the formof a “portion of an area” may cover between 0.1% and 100% of the area.As another example, sentences in the form of a “region on the user'sforehead” may cover between the smallest area captured by a single pixel(such as 0.1% or 5% of the forehead) and 100% of the forehead. The word“region” refers to an open-ended claim language, and a camera said tocapture a specific region on the face may capture just a small part ofthe specific region, the entire specific region, and/or a portion of thespecific region together with additional region(s).

Sentences in the form of “angle greater than 20°” refer to absolutevalues (which may be +20° or −20° in this example), unless specificallyindicated, such as in a phrase having the form of “the optical axis ofCAM is 200 above/below the Frankfort horizontal plane” where it isclearly indicated that the CAM is pointed upwards/downwards. TheFrankfort horizontal plane is created by two lines from the superioraspects of the right/left external auditory canal to the most inferiorpoint of the right/left orbital rims.

The terms “comprises,” “comprising,” “includes,” “including,” “has,”“having”, or any other variation thereof, indicate an open-ended claimlanguage that does not exclude additional limitations. The “a” or “an”is employed to describe one or more, and the singular also includes theplural unless it is obvious that it is meant otherwise; for example,sentences in the form of “a CAM configured to take thermal measurementsof a region (TH_(ROI))” refers to one or more CAMs that take thermalmeasurements of one or more regions, including one CAM that takesthermal measurements of multiple regions; as another example, “acomputer” refers to one or more computers, such as a combination of awearable computer that operates together with a cloud computer.

The phrase “based on” is intended to mean “based, at least in part, on”.Additionally, stating that a value is calculated “based on X” andfollowing that, in a certain embodiment, that the value is calculated“also based on Y”, means that in the certain embodiment, the value iscalculated based on X and Y.

The terms “first”, “second” and so forth are to be interpreted merely asordinal designations, and shall not be limited in themselves. Apredetermined value is a fixed value and/or a value determined any timebefore performing a calculation that compares a certain value with thepredetermined value. A value is also considered to be a predeterminedvalue when the logic, used to determine whether a threshold thatutilizes the value is reached, is known before start performingcomputations to determine whether the threshold is reached.

The embodiments of the invention may include any variety of combinationsand/or integrations of the features of the embodiments described herein.Although some embodiments may depict serial operations, the embodimentsmay perform certain operations in parallel and/or in different ordersfrom those depicted. Moreover, the use of repeated reference numeralsand/or letters in the text and/or drawings is for the purpose ofsimplicity and clarity and does not in itself dictate a relationshipbetween the various embodiments and/or configurations discussed. Theembodiments are not limited in their applications to the order of stepsof the methods, or to details of implementation of the devices, set inthe description, drawings, or examples. Moreover, individual blocksillustrated in the figures may be functional in nature and therefore maynot necessarily correspond to discrete hardware elements.

Certain features of the embodiments, which may have been, for clarity,described in the context of separate embodiments, may also be providedin various combinations in a single embodiment. Conversely, variousfeatures of the embodiments, which may have been, for brevity, describedin the context of a single embodiment, may also be provided separatelyor in any suitable sub-combination. Embodiments described in conjunctionwith specific examples are presented by way of example, and notlimitation. Moreover, it is evident that many alternatives,modifications, and variations will be apparent to those skilled in theart. It is to be understood that other embodiments may be utilized andstructural changes may be made without departing from the scope of theembodiments. Accordingly, this disclosure is intended to embrace allsuch alternatives, modifications, and variations that fall within thespirit and scope of the appended claims and their equivalents.

We claim:
 1. A wearable device, comprising: a frame configured to beworn on a user's head; and an inward-facing camera (camera) physicallycoupled to the frame; wherein the optical axis of the camera is eitherabove the Frankfort horizontal plane and pointed upward to capture animage of a region of interest (ROI) above the user's eyes, or theoptical axis is below the Frankfort horizontal plane and pointeddownward to capture an image of an ROI below the user's eyes; the cameracomprises a sensor and a lens; wherein the sensor plane is tilted bymore than 2° relative to the lens plane according to the Scheimpflugprinciple in order to capture a sharper image.
 2. The wearable device ofclaim 1, wherein the ROI covers first and second areas, and the tiltbetween the lens plane and the sensor plane is adjusted such that theimage of the first area is shaper than the image of the second area;whereby the first area is more important for detecting a physiologicalresponse of the user than the second area.
 3. The wearable device ofclaim 2, wherein the first area is on the user's upper lip and thesecond area is on the user's cheek, and the tilt is adjusted such thatthe image of the first area is shaper than the image of the second area.4. The wearable device of claim 2, wherein the first area is on theuser's upper lip and the second area is on the user's nose, and the tiltis adjusted such that the image of the first area is shaper than theimage of the second area.
 5. The wearable device of claim 2, wherein thefirst area is on the user's lips and the second area is on the user'schin, and the tilt is adjusted such that the image of the first area isshaper than the image of the second area.
 6. The wearable device ofclaim 2, wherein the camera is a visible-light camera, the first area ison the user's lower forehead (including an eyebrow) and the second areais on the user's upper forehead, and the tilt is adjusted such that theimage of the first area is shaper than the image of the second area. 7.The wearable device of claim 2, wherein the camera is a thermal camera,the first area is on the middle and upper part of the user's forehead(below the hair line), the second area is on the lower part of theuser's forehead, and the tilt is adjusted such that the image of thefirst area is shaper than the image of the second area.
 8. The wearabledevice of claim 1, wherein the ROI covers a first area on the cheekstraight above the upper lip, a second area on the cheek from the edgeof the upper lip towards the ear, and a third area on the nose; whereinthe tilt between the lens plane and the sensor plane is adjusted suchthat the image of the first area is shaper than both the images of thesecond and third areas.
 9. The wearable device of claim 1, wherein theROI covers both the region on the user's upper lip and a second regionon the body below the face when the user is standing; and wherein thesharpness of the image of both the region and the second region, whichis obtained from the camera when the user wears the frame, is betterthan the sharpness of an image of both the region and the second regionthat would be obtained from a similar camera having the same sensor andlens, but with the lens plane parallel to the sensor plane.
 10. Thewearable device of claim 1, wherein the camera is a thermal cameraconfigured to take thermal measurements of the ROI; and furthercomprising a computer configured to detect a physiological responsebased on the thermal measurements.
 11. The wearable device of claim 10,wherein the sensor comprises multiple sensing elements, and the computeris configured to process time series measurements of each sensingelement individually in order to detect the physiological response. 12.The wearable device of claim 10, wherein the thermal camera weighs below10 g, is located less than 10 cm from the user's face, and the tilt ofthe lens plane relative to the sensor plane is fixed; and wherein thefixed tilt is selected according to an expected orientation between thecamera and the ROI when the user wears the frame.
 13. The wearabledevice of claim 1, wherein the camera is a visible-light cameraconfigured to take visible-light images of the ROI; and furthercomprising a computer configured to generate an avatar of the user basedon the visible-light images.
 14. The wearable device of claim 1, whereinthe camera is a visible-light camera configured to take visible-lightimages of the ROI; and further comprising a computer configured todetect the user's emotional response based on identifying facialexpressions in the visible-light images.
 15. The wearable device ofclaim 1, wherein the camera is a light field camera configured to: (i)implement a predetermined blurring at a certain Scheimpflug angle, and(ii) decode the predetermined blurring as function of the certainScheimpflug angle.
 16. The wearable device of claim 1, furthercomprising an adjustable electromechanical tilting mechanism configuredto change the tilt of the lens plane relative to the sensor planeaccording to the Scheimpflug principle, based on the orientation betweenthe camera and the ROI when the camera is mounted on the user's head.17. The wearable device of claim 16, wherein the camera, including theadjustable electromechanical tilting mechanism, weighs less than 10 g,located less than 15 cm from the user's face, and the adjustableelectromechanical tilting mechanism is able to change the tilt betweenthe lens and sensor planes in a limited range below 30° between the twoutmost orientations between the lens and sensor planes.
 18. The wearabledevice of claim 16, wherein the camera, including the adjustableelectromechanical tilting mechanism, weighs less than 10 g, located lessthan 15 cm from the user's face, and the adjustable electromechanicaltilting mechanism is able to change the tilt between the lens and sensorplanes in a limited range below 20° between the two utmost orientationsbetween the lens and sensor planes.
 19. A device comprising: aninward-facing head-mounted camera (camera) configured to capture, whenworn on a user's head, an image of a region of interest (ROI) on atleast one of the following regions on the user's face: the forehead, thenose, the upper lip, a cheek, and the lips; and the camera comprises asensor and a lens; wherein the sensor plane is tilted by more than 2°relative to the lens plane according to the Scheimpflug principle inorder to capture a sharper image.
 20. The device of claim 19, whereinthe ROI covers first and second areas, and the tilt between the lensplane and the sensor plane is adjusted such that the image of the firstarea is shaper than the image of the second area, whereby the first areais more important for detecting a physiological response of the userthan the second area.